Click here to go to the forum index Click here for the home page
 
Author Message

<  Firmwares, enhancements and upgrades  ~  REQ: Patch for EPG cache flaw

Page 1 of 4
Goto page 1, 2, 3, 4  Next
dvbgeek
Posted: Fri Aug 03, 2007 6:11 pm Reply with quote
Frequent contributor Joined: 16 Apr 2007 Posts: 168 Location: Sweden
I've discovered a flaw in the Toppy's EPG harvesting which doesn't show in countries running a sane EIT implementation but affects all Toppys in Sweden and also EPG software (eit2mei) on the boxes that use API calls to get data from the native EPG.

Our DTT provider sends a crazy EIT stream that carries approx. 24 hours of data, but only a few hours of fulltext descriptions. As the box is turned on, the (built-in) EPG shows correct descriptions for all programmes (that is, the same information that is actually in the EIT stream at the moment).
If the box is left "on" for a few hours, the programmes that are approaching gets full descriptions in the EIT stream but the Toppy doesn't catch anything other than the Extended field (the overflow from the new longer description).

I have to presume the bug exists in all Topfield firmwares but only "shows" in countries which don't distribute full-text descriptions for all programmes that are in the EIT stream at any given time. (Sweden does not, as you can see here).

Let's see if I can explain this with an example description:

<Here's a short description >
<Here's the longer Short description. It contains more detailed information>
< and goes on into the Extended description field.>

Early in the day, presuming the theoretical show is due for later in the day, the box shows only:
Here's a short description

If the box is left "on" during the day, once the DTT provider has included the full description in the stream, the box will show (and also send to eit2mei):
Here's a short description__and goes on into the Extended description field. <-- there are TWO spaces (replaced by me with underscores to be more visible) between the old short and the extended descriptions as the initial short description ends in a space (the "fulltext short description" does not) and the extended always begins with one.

If at this point I reboot the box, it will pick up the correct information:
Here's the longer Short description. It contains more detailed information and goes on into the Extended description field.
But leaving it on will affect future programs in the same way as the EIT data changes during the day.

I think this may be a cache issue (since a reboot cures it temporarily) or possibly the built-in EIT harvester ignores short descriptions for programmes that already have them in memory, not bothering to check if the data has changed, and just captures the Extended information in addition to what it already has in memory.

In any event, curing this should be a matter of forcing the EPG cache to clear itself on each scan. Is this something that could be done through a firmware patch or would it require changes by Topfield (which I think is out of the question, they just upped the major version number for our Swedish firmware without fixing even one known bug, and just committing some minor changes that didn't even affect any issue encountered in Sweden).

Any takers?

_________________
Model: TF5700PVRt HDMI (ID 13426) / Firmware: 2.84 with recommended patches

Autoload TAPs: Power Down / TF5000Display / eit2mei - mei2archive - DescriptionExtender / MyStuff / TAP Commander (always the latest version of everything)
View user's profile Send private message
R2-D2
Posted: Fri Aug 03, 2007 6:40 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
bdb's probably your man on this, but when I had a look at this stuff I found it incredibly complicated, although it did seem like the 2 bits in the "valid bits" field of an event was populated in two passes: one for the short description and one for the extended info. I could (of course) be wrong, but this might be necessary due to the data stream format. Anyway, it may be that this is what you're seeing: the EventID is the same and there's a big gap between short and extended info being sent, but once the short description has been received it's not updated. So you eventually get the original short description + new extended description. A workaround might be possible in making the code update the short description when it receives an event that it thinks it's already got.

If you see what I mean. Smile
View user's profile Send private message Visit poster's website
dvbgeek
Posted: Sat Aug 04, 2007 11:30 am Reply with quote
Frequent contributor Joined: 16 Apr 2007 Posts: 168 Location: Sweden
R2-D2 wrote:
bdb's probably your man on this, but when I had a look at this stuff I found it incredibly complicated, although it did seem like the 2 bits in the "valid bits" field of an event was populated in two passes: one for the short description and one for the extended info. I could (of course) be wrong, but this might be necessary due to the data stream format. Anyway, it may be that this is what you're seeing: the EventID is the same and there's a big gap between short and extended info being sent, but once the short description has been received it's not updated. So you eventually get the original short description + new extended description. A workaround might be possible in making the code update the short description when it receives an event that it thinks it's already got.

If you see what I mean. Smile

Yes, that's exactly what I think is happening as well.
Because the short description changes, and the extended field is empty or non-existant (the Toppy says "No extended information") until the "short Short description" is replaced by the "longer Short description" (which often overflows into the Extended field thereby populating it), the Toppy just sticks the "short Short description" in it's memory and when the Extended information arrives for the same EventID, the Toppy just takes the Extended info and doesn't update the Short description because it thinks it already has it (it doesn't include provisions for the possibility that the DTT provider might have implemented such a stupid EPG update scheme).
The solution, as you said, would of course be to have the box always clear the "valid bit" for the Short description for any EventID it already has in memory, on every scan.

The question, of course, is if this can be done through a simple patch to the existing firmware (or if such patching could introduce undesirable behaviour by clearing this bit), or if it requires deeper changes that require access to the actual source code of the firmware (which SHOULD be available seeing as how they use code from several GPL'd projects)?

_________________
Model: TF5700PVRt HDMI (ID 13426) / Firmware: 2.84 with recommended patches

Autoload TAPs: Power Down / TF5000Display / eit2mei - mei2archive - DescriptionExtender / MyStuff / TAP Commander (always the latest version of everything)
View user's profile Send private message
R2-D2
Posted: Tue Aug 14, 2007 2:10 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
If you can connect to the Toppy via serial then have a stab at this command to see if it improves anything (for TF5700HDMI (13426) 3.00 only):
Code:
tuow 1122eeff 8016978c 0
I think this might suppress the check on whether the short description has been seen. But I could be wide of the mark..
View user's profile Send private message Visit poster's website
dvbgeek
Posted: Tue Aug 14, 2007 4:48 pm Reply with quote
Frequent contributor Joined: 16 Apr 2007 Posts: 168 Location: Sweden
I don't have a suitable serial cable at the moment, but if I do manage to find one, is this a command I'm supposed to enter through a terminal emulator (like Putty) and if so, what settings do I need to enter for the serial connection in the terminal emulator?

Also, I do hope this is a command that affects only the runtime firmware in memory and not the actual on-flash firmware... ?

_________________
Model: TF5700PVRt HDMI (ID 13426) / Firmware: 2.84 with recommended patches

Autoload TAPs: Power Down / TF5000Display / eit2mei - mei2archive - DescriptionExtender / MyStuff / TAP Commander (always the latest version of everything)
View user's profile Send private message
R2-D2
Posted: Tue Aug 14, 2007 5:12 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
dvbgeek wrote:
Also, I do hope this is a command that affects only the runtime firmware in memory and not the actual on-flash firmware... ?

Well, yes, but not that it would make a huge difference. Smile I've packaged it up as a quick TAP. Still not sure it will make much difference to your EPG problems, though.
View user's profile Send private message Visit poster's website
dvbgeek
Posted: Tue Aug 14, 2007 8:07 pm Reply with quote
Frequent contributor Joined: 16 Apr 2007 Posts: 168 Location: Sweden
R2-D2 wrote:
dvbgeek wrote:
Also, I do hope this is a command that affects only the runtime firmware in memory and not the actual on-flash firmware... ?

Well, yes, but not that it would make a huge difference. Smile I've packaged it up as a quick TAP. Still not sure it will make much difference to your EPG problems, though.

Thanks, I'll try it out and let you know what happens.

_________________
Model: TF5700PVRt HDMI (ID 13426) / Firmware: 2.84 with recommended patches

Autoload TAPs: Power Down / TF5000Display / eit2mei - mei2archive - DescriptionExtender / MyStuff / TAP Commander (always the latest version of everything)
View user's profile Send private message
andyrogers
Posted: Tue Aug 14, 2007 8:41 pm Reply with quote
Frequent contributor Joined: 07 Dec 2005 Posts: 814
R2-D2

Along the lines of patches for the EPG. I see that from the following EPG patch http://www.tapworld.net/content/view/382/42/ their is one available that reduces the Extended memory thus freeing up some spare memory.

Do you know if a similar patch could be made for the 5800 for the people who do not use the extended EPG?

Thanks

Andy

_________________

Firmware: 5.14.09 Patched
AutoStart Taps: MyStuff v5.52b, eit2mei beta 7.8m7, , Power Manager v1.2, Tap Commander v1.32, TF5000 Display v1.53a, Discription Extender v2.3, SDS 1.3b, QuickJump 1.72
Other Taps: Sudoka, mei2archive, mei2eit, snake, meisearch
View user's profile Send private message
R2-D2
Posted: Tue Aug 14, 2007 9:53 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
andyrogers wrote:
Do you know if a similar patch could be made for the 5800 for the people who do not use the extended EPG?

The buffer is combined in recent TF5800 firmwares (3.5Mb, I think). So the benefit will depend on what EPG method you're using. This patch will drop the buffer to 0.5Mb, I think.
View user's profile Send private message Visit poster's website
bdb
Posted: Tue Aug 14, 2007 9:58 pm Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
Not relevent (apart for the few; notably you Rolling Eyes who have not yet progressed as far as 12.88.

see http://forum.toppy.org.uk/forum/viewtopic.php?p=95241

bdb

edit:
oops; beaten to it and sorry, mixed you up with Andy K Embarassed
View user's profile Send private message
andyrogers
Posted: Tue Aug 14, 2007 10:28 pm Reply with quote
Frequent contributor Joined: 07 Dec 2005 Posts: 814
R2-D2 wrote:
andyrogers wrote:
Do you know if a similar patch could be made for the 5800 for the people who do not use the extended EPG?

The buffer is combined in recent TF5800 firmwares (3.5Mb, I think). So the benefit will depend on what EPG method you're using. This patch will drop the buffer to 0.5Mb, I think.


Thanks R2-D2 that was very quick.

I only use the freeview epg to populate my toppy, so do not use the extended memory really.

I shall let you know how i get on.

Andy

_________________

Firmware: 5.14.09 Patched
AutoStart Taps: MyStuff v5.52b, eit2mei beta 7.8m7, , Power Manager v1.2, Tap Commander v1.32, TF5000 Display v1.53a, Discription Extender v2.3, SDS 1.3b, QuickJump 1.72
Other Taps: Sudoka, mei2archive, mei2eit, snake, meisearch
View user's profile Send private message
Andy K
Posted: Tue Aug 14, 2007 10:31 pm Reply with quote
Frequent contributor Joined: 14 Jun 2005 Posts: 3520
Laughing

I did make it as far as 12.88 twice but keep coming back.

The main reason IIRC was poor remote response.
I was also not impressed with the half attempt to fix the CYR

_________________
Autostart TAPs: Jags 3, Bookmark 2uk, Quickjump 1.71, Power Manager 1.1, Description Extender 1.5/2.1, MEI2Archive 3.8l6, Tap Launcher 3.5a, Tap Commander, AccurateBMExtend 0.3, RemoteExt 1.5, TunerRecAR.2
Launched during EPG scan: Crid, SeriesLink 0.35
TF5800 Version 5.13.65 PHT2UFXp5Xw3RpPcE2Bf2BqRsRh3Pf1Ec2ErEfHe1 Ra3Cf2Ct
Samsung 400Gb+Fan
View user's profile Send private message
andyrogers
Posted: Tue Aug 14, 2007 10:33 pm Reply with quote
Frequent contributor Joined: 07 Dec 2005 Posts: 814
R2-D2

Iam trying to install this patch onto 5.13.59 but it won't let me Confused .

Andy

_________________

Firmware: 5.14.09 Patched
AutoStart Taps: MyStuff v5.52b, eit2mei beta 7.8m7, , Power Manager v1.2, Tap Commander v1.32, TF5000 Display v1.53a, Discription Extender v2.3, SDS 1.3b, QuickJump 1.72
Other Taps: Sudoka, mei2archive, mei2eit, snake, meisearch
View user's profile Send private message
R2-D2
Posted: Tue Aug 14, 2007 11:25 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
andyrogers wrote:
Iam trying to install this patch onto 5.13.59 but it won't let me Confused .

Hey! Did I mention I was new to this? Smile I tried doing it the way the other EPG patch did the generic patch, and it worked in TFPCompiler... but it doesn't work in FWPatcher. I'm sure FireBird has explained why, but I forget. Anyway, done it the way I've done it before and it worked OK. (Plus I added 5.13.59 to the specific patches, which helps.)

Not sure you're going to enjoy a 0.5Mb buffer, though, if you're using mei2eit. Smile Something like MyStuff + a MyStuff.mei from rt2mei might work, though.
View user's profile Send private message Visit poster's website
andyrogers
Posted: Wed Aug 15, 2007 8:31 pm Reply with quote
Frequent contributor Joined: 07 Dec 2005 Posts: 814
Hi R2-D2

THanks for the patch for 5.13.59 last nightm it oes work, but 0.5mb seems to be not enough. this mornign I had a lot of "Unknown" names in the EPg table.

Could you increase it 1.5mb and I can see how i get on.

Thanks

Andy

_________________

Firmware: 5.14.09 Patched
AutoStart Taps: MyStuff v5.52b, eit2mei beta 7.8m7, , Power Manager v1.2, Tap Commander v1.32, TF5000 Display v1.53a, Discription Extender v2.3, SDS 1.3b, QuickJump 1.72
Other Taps: Sudoka, mei2archive, mei2eit, snake, meisearch
View user's profile Send private message

Display posts from previous:  

All times are GMT + 1 Hour
Page 1 of 4
Goto page 1, 2, 3, 4  Next

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum