Beirdobot, irc.freenode.net

Monday, October 30th, 2017, 00:41 UTC

[00:41:35]

dekarl (dekarl!~dekarl@mythtv/developer/dekarl) has quit (Ping timeout: 240 seconds)

[00:41:39]

dekarl1 (dekarl1!~dekarl@mythtv/developer/dekarl) has joined #mythtv

[03:07:02]

peper03_ (peper03_!~peper03@mythtv/developer/peper03) has joined #mythtv

[03:10:23]

peper03 (peper03!~peper03@mythtv/developer/peper03) has quit (Ping timeout: 248 seconds)

[05:01:47]

MythNotifyBot:

[15mizar.mythtv.org] HTTP Smolt 3OK => 5CRITICAL || 7CRITICAL – Socket timeout after 10 seconds

[06:55:44]

pccjjsetby (pccjjsetby!~hi@cpe-67-240-33-252.nycap.res.rr.com) has joined #mythtv

[06:55:47]

pccjjsetby (pccjjsetby!~hi@cpe-67-240-33-252.nycap.res.rr.com) has quit (Remote host closed the connection)

[07:25:09]

SteveGoodey (SteveGoodey!~steve@host109-149-25-3.range109-149.btcentralplus.com) has joined #mythtv

[07:29:41]

gregl (gregl!~greg@cpe-66-67-122-101.nycap.res.rr.com) has quit (Ping timeout: 240 seconds)

[07:42:33]

gregl (gregl!~greg@cpe-66-67-122-101.nycap.res.rr.com) has joined #mythtv

[08:06:23]

sphery (sphery!~mdean@mythtv/developer/sphery) has quit (Ping timeout: 248 seconds)

[08:23:47]

Merlin83b (Merlin83b!~Daniel@office.34sp.com) has joined #mythtv

[08:49:08]

willcooke (willcooke!~willcooke@host-92-18-125-159.as13285.net) has joined #mythtv

[08:49:09]

willcooke (willcooke!~willcooke@host-92-18-125-159.as13285.net) has quit (Changing host)

[08:49:09]

willcooke (willcooke!~willcooke@ubuntu/member/willcooke) has joined #mythtv

[08:54:50]

stuarta:

yay. it worked again

[08:55:38]

MythNotifyBot:

[15mizar.mythtv.org] HTTP Smolt 5CRIT => 3OK || 7HTTP OK: HTTP/1.1 200 OK – 2368 bytes in 0.408 second response time

[09:45:29]

stuarta:

tgm4883: do you have an account in trac?

[09:51:43]

sphery (sphery!~mdean@mythtv/developer/sphery) has joined #mythtv

[10:09:18]

SteveGoodey (SteveGoodey!~steve@host109-149-25-3.range109-149.btcentralplus.com) has quit (Quit: Konversation terminated!)

[11:10:27]

willcooke (willcooke!~willcooke@ubuntu/member/willcooke) has quit (Ping timeout: 240 seconds)

[11:10:56]

willcooke (willcooke!~willcooke@host-92-18-125-159.as13285.net) has joined #mythtv

[11:10:56]

willcooke (willcooke!~willcooke@ubuntu/member/willcooke) has joined #mythtv

[11:10:56]

willcooke (willcooke!~willcooke@host-92-18-125-159.as13285.net) has quit (Changing host)

[11:49:34]

gregl (gregl!~greg@cpe-66-67-122-101.nycap.res.rr.com) has quit (Quit: Leaving)

[12:23:44]

SteveGoodey (SteveGoodey!~steve@host109-149-25-3.range109-149.btcentralplus.com) has joined #mythtv

[12:31:10]

ooshlablu (ooshlablu!~ooshlablu@2601:18d:4600:5f31:7d31:4040:c72a:272e) has quit (Remote host closed the connection)

[12:49:21]

markspieth|2 (markspieth|2!~markspiet@CPE-124-188-72-98.facz1.ken.bigpond.net.au) has joined #mythtv

[12:51:24]

markspieth (markspieth!~markspiet@CPE-124-188-72-98.facz1.ken.bigpond.net.au) has quit (Ping timeout: 248 seconds)

[13:16:01]

stuarta:

please excuse the noise, upgrading sso

[13:20:37]

stuarta:

oh i sneaked it in before monitoring tripped :)

[13:44:39]

ooshlablu (ooshlablu!~ooshlablu@50.234.38.126) has joined #mythtv

[13:48:28]

tgm4883:

stuarta: No I don't think so

[13:55:42]

Spicy-Rabbit (Spicy-Rabbit!~Jordack@75-151-31-172-Michigan.hfc.comcastbusiness.net) has joined #mythtv

[14:12:26]

stuarta:

tgm4883: didn't think so, i wanted to assign you the packaging ticket and couldn't find you ;-)

[14:12:51]

peper03_:

Hi guys! I'm looking (yet again) at how to add a quick scan of all Freesat channels including (if desired) assigning the correct channel numbers. I keep coming back to this but can usually only snatch an hour or two here and there, so it's not happening quickly :(

[14:13:15]

stuarta:

peper03_: what did you have in mind?

[14:13:25]

peper03_:

There are several private data descriptors which contain the mappings between regions, channel numbers and categories.

[14:13:53]

peper03_:

I've finally got those relationships more or less figured out but I'm getting stuck on how to add that to our source code

[14:14:14]

peper03_ is now known as peper03

[14:14:33]

stuarta:

oh cool

[14:14:57]

stuarta:

i'd start by building out the parsing of those descriptors in the various mpeg* dvb* classes

[14:15:22]

peper03:

That's what I was just looking at.

[14:15:46]

stuarta:

it's fairly trivial, and should fit inside your 1–2hr slots ;-)

[14:16:12]

peper03:

Let me explain the overall structure of the data as transmitted:

[14:16:47]

peper03:

Multiple BATs are transmitted, each one representing a different "type/region"

[14:17:04]

stuarta:

ah yes, we have no BAT parsing what so ever

[14:17:10]

peper03:

England SD, England G2 and England HD

[14:17:17]

peper03:

Then the same for Wales, NI and Scotland.

[14:17:20]

stuarta:

well, we have the basics

[14:17:31]

stuarta:

yep

[14:17:35]

peper03:

Each of those BATs is made up of multiple parts.

[14:17:53]

stuarta:

multiple sections, yes that's expected, the code should re-assemble that for you

[14:18:39]

peper03:

For each BAT (let's assume all the sections have been combined), there is another table listing regions.

[14:18:55]

stuarta:

transmitted in the BAT?

[14:19:01]

peper03:

Yes.

[14:19:17]

stuarta:

cool

[14:19:27]

peper03:

There are four different descriptors for all the data.

[14:20:11]

peper03:

I think only the "England *" bouquets have multiple regions, but that's not a given.

[14:20:57]

peper03:

Then there are more tables for the channel numbers. Depending on the region you choose, you get a different line-up.

[14:21:26]

stuarta:

here's an example pulled from a previous scan i did https://paste.fedoraproject.org/paste/32r3vgAtao2ir8zlI3TwaQ

[14:21:46]

peper03:

Finally there is a table of categories (again linked to the regions), and a final table listing which channels belong to which categories.

[14:21:53]

stuarta:

0x5f = FSAT

[14:22:07]

stuarta:

0xd4–0xd7 for the extra tables

[14:22:23]

peper03:

Correct. They actually go to 0xd8

[14:22:36]

** stuarta hunts the logs **

[14:23:16]

peper03:

I've not figured out what 0xd6 and 0xd7 are used for. They're short and always the same.

[14:24:23]

stuarta:

ah yes, there's a 0xd8

[14:24:48]

stuarta:

https://paste.fedoraproject.org/paste/6J0xDsfRz2nU4saV6tELCg

[14:25:19]

stuarta:

note that the BAT dumping gets truncated by our log printing

[14:25:27]

stuarta:

will have to work out how to fix that

[14:26:30]

peper03:

0xd8 is the category list

[14:26:48]

stuarta:

agreed

[14:27:41]

peper03:

And bouquet 0x111 is "Scotland HD"

[14:27:56]

peper03:

So that's the category list for Scotland HD

[14:28:10]

peper03:

For whatever reason they're not always the same

[14:28:27]

stuarta:

found a few more, with variations in bouquet id's

[14:29:03]

peper03:

So, we have to parse multiple multi-section BATs to get the high-level regions, and then start working from there.

[14:30:13]

peper03:

Trying to keep all the relationships in my head and then work out how that can fit into our code is making my head hurt :(

[14:30:16]

stuarta:

do we "require" the category BAT? actually, out of all of them, which descriptors do we need??

[14:30:39]

stuarta:

tbh, we use the category for a program from the guide data

[14:30:49]

peper03:

No, we don't require it.

[14:30:52]

stuarta:

so you could easily ignore that one

[14:31:07]

stuarta:

0xd4–0xd7 all seem to be transmitted together

[14:31:27]

peper03:

Category is the wrong word. Genre or group would be better.

[14:31:33]

stuarta:

true

[14:31:44]

stuarta:

still, we don't need it

[14:31:58]

stuarta:

so that in theory leaves us with just the one BAT to process

[14:32:14]

peper03:

Theoretically we could still use it to automatically set up channel groups if desired but it's not top of my list.

[14:32:25]

peper03:

No, you still need to process all the BATs.

[14:32:26]

stuarta:

what else would you do with it?

[14:32:54]

peper03:

Not much else. The information is there so it could be done, but I probably wouldn't use it myself.

[14:33:27]

peper03:

One thing it could theoretically be used for is to say "please mark all home shopping channels as invisible".

[14:33:33]

stuarta:

this is how i forsee it, please say if you think differently

[14:33:40]

stuarta:

1) do channel scan, collecting all BAT's

[14:33:41]

peper03:

But that starts getting into a different problem.

[14:33:58]

stuarta:

2) ask which region people want

[14:34:07]

stuarta:

3) mark everything else as invisible

[14:34:29]

stuarta:

or were you thinking to look for just BAT's initially to choose a region?

[14:34:53]

sphery (sphery!~mdean@mythtv/developer/sphery) has quit (Ping timeout: 252 seconds)

[14:35:44]

stuarta:

i mean, i'd love to see us use the bat properly

[14:36:27]

peper03:

First, I would try to collect all the data and parse it into internal structures. Then we get the UI side of things where we could let the user drill down to the exact configuration they want.

[14:37:18]

stuarta:

yeah, all of the BAT's would be collected by the time you finish walking the mplex's

[14:38:03]

peper03:

Another thing that's not clear to me is how to know when all BATs have been found? The sections are easy enough but I've not found anywhere that would let us know how many BATs to look for in the first place.

[14:38:22]

peper03:

All the information is there on any one transponder, I think.

[14:38:34]

peper03:

That's why it would be nice and fast.

[14:39:09]

stuarta:

you never know what the full set of BAT's is, until such time as they start repeating themselves

[14:39:33]

stuarta:

and there is a broadcast interval for BAT's, it's either 10 or 30 seconds.

[14:40:07]

peper03:

OK, so each BAT must be sent at least once ever 10/30 seconds?

[14:40:33]

peper03:

s/ever/every/

[14:40:34]

** stuarta is digging through the relevant docs to find the answer **

[14:46:57]

stuarta:

10s for BAT

[14:47:45]

peper03:

Is that 10s for any one BAT, or for all BATs?

[14:47:58]

stuarta:

did you see my PM?

[14:48:09]

peper03:

Yes, I'm just trying to interpret the wording.

[14:48:25]

peper03:

"All sections of *the* BAT"

[14:48:52]

stuarta:

interpretation is half the battle

[14:49:31]

peper03:

Although, I suppose it must apply to all BATs. If any one BAT must be completely transmitted at least once every ten seconds, they all must be.

[14:49:46]

stuarta:

you could take that to mean either 1) the BAT relating to this mux (which doesn't really make sense, as a single mux can belong to multiple BAT's), or 2) all BAT's, and each BAT must be sent every 10s

[14:50:51]

stuarta:

it *should* be number 2, inline with the conclusion you came to

[14:51:43]

peper03:

OK, so basically scan for 10 seconds and discard any duplicates.

[14:54:30]

stuarta:

there will be a version number within the BAT, which will be incremented if the BAT is updated

[14:57:24]

peper03:

OK, so it looks like I should start in MPEGStreamData::IsRedundant and also add support for BATs there.

[14:57:54]

peper03:

Ah, no. That's in DVBStreamData...

[14:58:25]

stuarta:

there is already some basic BAT handling

[14:59:00]

stuarta:

you are really looking at a higher level, pulling the cached BAT's and getting info out

[15:00:53]

peper03:

I already started looking at BouquetAssocationTable::Parse but it didn't seem like that was being called with all the data.

[15:01:11]

peper03:

Are the separate sections combined together at some point already?

[15:02:23]

sphery (sphery!~mdean@mythtv/developer/sphery) has joined #mythtv

[15:02:33]

stuarta:

yes, the channel scanner users them, if only to dump them out for inspection

[15:02:57]

stuarta:

actually, i'll rephrase

[15:03:06]

stuarta:

we see them, and dump them out as we get them

[15:03:19]

stuarta:

iirc, that's in dvbtables, or dvbstreamdata

[15:03:47]

stuarta:

i don't think the channel scanner actually uses them specifically

[15:04:32]

peper03:

I was in ChannelScanSM::HandleBAT already too.

[15:04:53]

stuarta:

ok, so that'll be where any action happens

[15:05:03]

peper03:

So maybe combine them there?

[15:05:44]

peper03:

I'm doing to my best to learn all the different "standard" relationships between the different DVB data structures but I have to admit it's making my head spin a bit.

[15:06:04]

peper03:

Knowing then where the best place is in our code for new functionality is another step again.

[15:07:25]

stuarta:

so it's parsed here https://github.com/MythTV/mythtv/blob/master/ . . . les.cpp#L165

[15:09:12]

stuarta:

although that's very low level

[15:10:10]

peper03:

Yes, that's where I started.

[15:11:02]

stuarta:

by the time you get to ChannelScanSM::HandleBAT, you have a complete BouquetAssociationTable

[15:12:34]

stuarta:

https://github.com/MythTV/mythtv/blob/master/ . . . ables.h#L179

[15:12:40]

stuarta:

that's also a good point to add stuff

[15:13:10]

stuarta:

like an isFreeSat() method

[15:14:21]

stuarta:

you can also enhance the ::Parse method to decode the new descriptors into local data inside a BouquetAssociationTable object

[15:14:45]

stuarta:

which would probably actually be a good idea

[15:15:17]

stuarta:

since when the constructor is called (from the lower level code which assembles the sections), ::Parse is called

[15:16:02]

peper03:

From what I can see ChannelScanSM::HandleBAT gets each section separately. I extended BouquetAssociationTable::toString to include the section number/total sections and I only see individual sections.

[15:16:21]

stuarta:

hmm, okay, then that's an issue we need to fix

[15:16:53]

gregl (gregl!~greg@cpe-66-67-122-101.nycap.res.rr.com) has joined #mythtv

[15:16:54]

stuarta:

please push the change to include the section number to the logging, then i can rerun to get current data

[15:17:06]

peper03:

Sure.

[15:17:32]

stuarta:

i've done that for other bits before, it makes deciphering this stuff much easier

[15:18:43]

stuarta:

now that does explain why i see some BAT's with tables 0xd4–0xd7, and then some with 0xd8. most likely different sections of the same table

[15:18:52]

peper03:

From what I can see, it doesn't look at anything much is checking the section number for any table.

[15:19:08]

peper03:

The only things I can find are caching and debug outputs.

[15:19:09]

stuarta:

yeah we do. iirc SDT's

[15:19:39]

peper03:

The caching only seems to be used to determine whether that section has already been seen.

[15:19:56]

peper03:

I don't see where it's being used to combine the sections together.

[15:19:57]

sphery (sphery!~mdean@mythtv/developer/sphery) has quit (Ping timeout: 260 seconds)

[15:21:33]

sphery (sphery!~mdean@mythtv/developer/sphery) has joined #mythtv

[15:24:00]

stuarta:

peper03: look in dvbstreamdata.cpp for the HasCachedAll* methods

[15:24:08]

stuarta:

looks like PAT & NIT right now

[15:24:29]

stuarta:

sorry SDT & NIT, PAT is in mpegstreamdata.cpp

[15:26:22]

peper03:

Ah, my bad. GetCachedNIT etc.

[15:29:07]

peper03:

OK, so extend that stuff for the BATs, and then process them in ChannelScanSM::GetChannelList...

[15:29:14]

stuarta:

pretty much

[15:29:35]

stuarta:

the channel scanner will have a call somewhere to HasCachedAll*

[15:30:35]

stuarta:

to which you'll have to add a check for having a complete BAT if and only if any partial BAT has been seen

[15:30:43]

stuarta:

(since it's optional)

[15:30:59]

stuarta:

eg, it's not on freeview, but is on freesat

[15:32:29]

peper03:

Do you mean there are no BATs on Freeview?

[15:32:57]

peper03:

(I only have sat)

[15:33:21]

stuarta:

nope

[15:33:47]

peper03:

No, there are no BATs on Freeview, or no, that's not what you mean?

[15:33:52]

stuarta:

if you think about it, it only makes sense on something like satellite which covers a wide geographical area

[15:33:58]

stuarta:

no bat's on freeview

[15:34:02]

peper03:

Ah :)

[15:34:07]

stuarta:

biab ~30mins

[15:34:15]

peper03:

np

[15:52:50]

stuarta:

back

[16:21:05]

peper03:

me too :)

[16:21:15]

stuarta:

time for a rescan with the new code

[16:24:52]

stuarta:

Section (11) Last Section (16) IsCurrent (1) <- BAT's are big

[16:25:05]

peper03:

In principle we could/should extend the code in MPEGDescriptor::toString but really it needs more context. Each descriptor is output individually but it really needs to know whether a private data specifier descriptor has been seen before to be sure it is decoding the data correctly.

[16:25:10]

stuarta:

admittedly that's a BSkyB one

[16:26:05]

peper03:

So far I've only been looking at the Freesat transponders.

[16:28:05]

stuarta:

there is some Bouquet stuff in dvbdescriptors.*

[16:29:26]

stuarta:

not much mind

[16:31:11]

peper03:

Yeah, we need something like 'FreesatLCNDescriptor' for 0xD3 that then decodes that particular blob. Then the same for the other descriptors.

[16:31:35]

stuarta:

i think it still belongs up in dvbdescriptor

[16:31:36]

stuarta:

i think it still belongs up in dvbdescriptors

[16:31:59]

stuarta:

even if the define itself goes down in the mpeg classes

[16:32:25]

peper03:

Yes, they would be. Just like DVBLogicalChannelDescriptor, DVBContentIndentifierDescriptor etc.

[16:33:16]

stuarta:

here i would think -> https://github.com/MythTV/mythtv/blob/master/ . . . ptors.h#L175

[16:33:39]

peper03:

Yep, that's where I've added them so far :)

[16:33:47]

stuarta:

excellent :)

[16:34:30]

peper03:

For decoding the blobs themselves, since they are private descriptors, the only way (as I understand it) to safely decode them is by seeing a "Private Data Specifier Descriptor" previously with the correct value.

[16:34:57]

peper03:

i.e. 0x46534154 (== "FSAT")

[16:35:37]

stuarta:

yeah, descriptor 0x5f == "FSAT"

[16:35:53]

stuarta:

that can basically be your isFreeSat() method

[16:35:57]

stuarta:

for the BAT

[16:38:08]

peper03:

I'll see what I can do :)

[16:38:53]

stuarta:

nice

[16:43:52]

stuarta:

looking at my new logs, it's definitely seeing a snapshot of the BAT's, and we aren't collecting them all

[16:46:31]

stuarta:

as we aren't waiting specifically for them, we merely print out what we see whilst waiting for the other tables

[16:49:20]

peper03:

I guess some additional logic in ChannelScanSM::UpdateChannelInfo to delay setting "transport_tune_complete" would help there.

[16:50:53]

stuarta:

something along the lines of if(CachedAnyBAT() && CachedAllBAT())

[16:51:06]

stuarta:

ie. if we have seen any of it, wait for all of it

[16:51:20]

peper03:

Something like that, yes.

[16:53:14]

stuarta:

so the 0x5f descriptor, also appears with the transport descriptors in the BAT

[16:54:01]

peper03:

Yes. That seems to be reliable and matches the general DVB specification as I understand it.

[16:54:08]

stuarta:

oh good

[16:54:36]

peper03:

But it *does* mean that it's not possible to reliably decode a private descriptor completely in isolation.

[16:54:55]

stuarta:

i see it in 2 places at lease

[16:54:58]

stuarta:

*least

[16:55:07]

stuarta:

BAT -> 0x57

[16:55:25]

peper03:

So the code in MPEGDescriptor::toString() really needs more context.

[16:55:25]

stuarta:

BAT -> Transports -> Transport_X -> 0x57

[16:55:51]

stuarta:

the toString part of this should be up in the BAT::toString

[16:57:19]

stuarta:

sheesh, it's also under BAT -> Transports -> 0x5f

[16:57:20]

peper03:

It is, but it calls MPEGDescriptor::toString for each of the descriptors found.

[16:57:25]

stuarta:

s/0x57/0x5f

[16:58:51]

peper03:

There's also a 0xd9 descriptor in the SDT, I see. Looks like it just contains language/short name.

[16:59:57]

stuarta:

yep

[17:04:38]

stuarta:

right, i'm off

[17:48:37]

Steve-Goodey (Steve-Goodey!~steve@host109-149-25-3.range109-149.btcentralplus.com) has joined #mythtv

[17:53:30]

Merlin83b (Merlin83b!~Daniel@office.34sp.com) has quit (Quit: Leaving)

[18:34:14]

gigem (gigem!~david@mythtv/developer/gigem) has quit (Quit: WeeChat 1.9.1)

[18:39:00]

gigem (gigem!~david@47.183.233.83) has joined #mythtv

[18:39:00]

gigem (gigem!~david@47.183.233.83) has quit (Changing host)

[18:39:00]

gigem (gigem!~david@mythtv/developer/gigem) has joined #mythtv

[18:43:01]

natanojl (natanojl!~jonatan@mythtv/developer/natanojl) has joined #mythtv

[18:43:38]

gregl (gregl!~greg@cpe-66-67-122-101.nycap.res.rr.com) has quit (Quit: Leaving)

[18:58:26]

gregl (gregl!~greg@cpe-66-67-122-101.nycap.res.rr.com) has joined #mythtv

[20:39:43]

Spicy-Rabbit (Spicy-Rabbit!~Jordack@75-151-31-172-Michigan.hfc.comcastbusiness.net) has quit (Quit: F This Day)

[20:48:06]

ooshlablu (ooshlablu!~ooshlablu@50.234.38.126) has quit (Remote host closed the connection)

[21:06:30]

Steve-Goodey (Steve-Goodey!~steve@host109-149-25-3.range109-149.btcentralplus.com) has quit (Quit: Konversation terminated!)

[21:14:31]

SteveGoodey (SteveGoodey!~steve@host109-149-25-3.range109-149.btcentralplus.com) has quit (Quit: Konversation terminated!)

[22:10:08]

amessina (amessina!~amessina@unaffiliated/amessina) has joined #mythtv

[22:13:35]

natanojl (natanojl!~jonatan@mythtv/developer/natanojl) has quit (Ping timeout: 240 seconds)

[22:45:29]

ooshlablu (ooshlablu!~ooshlablu@2601:18d:4600:5f31:91ef:7821:9310:efdc) has joined #mythtv

[23:16:57]

Tobbe5178 (Tobbe5178!~asdf@2001:2002:51eb:d24e:d87b:3d9f:7ec7:8695) has quit (Read error: Connection reset by peer)

MythLogBot@irc.freenode.net :: #mythtv

Current users (56):