Click here to go to the forum index Click here for the home page
 
Author Message

<  TAP and patch development  ~  How to save or access mheg5/mhp DSMCC files ?

Page 3 of 4
Goto page Previous  1, 2, 3, 4  Next
fblasot
Posted: Thu Oct 14, 2010 7:03 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Good news !
In the last 3 days I let my dsmccc tap start by timer at 6 am with option to log to file in append mode.
Now I checked the log and found that all the 3 times it downloads the necessary files in 5 minutes !!!!
Really I didn't expect this...
All the times I tried to run it live it tooks about 1 hour.
When I'm in live the time generally is in the evening (but I have seen with dvbstreamexplorer that data are always transmitted) and, that's may be the big difference, I have my pc connected both usb (with altair but not in turbo mode) and serial !
Sure the external connection slow process...
For the normal use it very very ok...(next days I will continue to monitor...).
Now I must pass to unzip using a tap my d0.zip ... d6.zip...

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
bdb
Posted: Thu Oct 14, 2010 8:47 pm Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
zlib:
I don't think I did anything unusual.
I have a zlib121.zip; seems to be 2003 vintage.
I added a header file to zutil.c to include my tap.h (which has some #defines to map stdio functions to the toppy flavour; for printf/fopen/fwrite/etc).
I removed stdlib.h from zutil.h
I then added .o files to my makefile until it stopped complaining.
I now have:
OBJS+=adler32.o compress.o crc32.o deflate.o inflate.o inffast.o inftrees.o trees.o uncompr.o zutil.o
I uncommented your #define uncompress

Quote:
I don't understand the difference between the two situation...

on the toppy; with the libdsmcc enabled, it took about a minute to receive and process 150KB of dsmcc data. Our MHEG data stream consists of several thousand very small files - unlike yours which has a few much larger files. As each dsmcc section is received, libdsmcc will write any completed file to the disk. During this period, the toppy pretty much seized up, as it was so busy - of the ~2MB of data it should have received it lost 90%; but was still able to extract lots of good files.

If the stream consists of a few larger files (firmware updates for example - not that we get any for the toppy...), libdsmcc needs to receive every section, before it can write out the file. If any sections are lost, it may either start again, or at least have to wait for that section to be rebroadcast. If you split the capture from the processing, then this restart/retry should not occur.

On the PC; I just took the 5MB dsmcc.data file that the toppy had saved (when I had the decoding disabled), wrote a simple wrapper to pass this file to libdsmcc, and was rewarded with lots of MHEG files on my PC disk.

bdb
View user's profile Send private message
fblasot
Posted: Fri Oct 15, 2010 10:59 am Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Thanks, now I understood.

No comments on my last post (Thu Oct 14, 2010 7:03 pm)...
Maybe you miss it ?

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
fblasot
Posted: Sun Nov 21, 2010 9:37 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Hi again !
finally I got my tap working:
- at 6 am (using the timer) it downloads the epg files from DSMCC in less of 10 minutes and unzip them (they are 7 zip files that produce 480 singles files where each file is 1 service and has the events of 1 day...)
- at each power on (not from the previous timer) it reads all the events files and fill the toppy epg.

The last point took 5/6 minutes, and I'm searching if it can be reduced.

To delete all the current event I do it in a very simple way:
Code:

void DeleteAllEpg(void)
{
int tvCount = 0, radioCount = 0;
int svcNum, eventNum;
int i;
TYPE_TapEvent *evtInfo;

    DEBUG("UpdateEpg: delete all Epg event !");

    TAP_Channel_GetTotalNum(&tvCount, &radioCount);
    for (svcNum=0;svcNum<tvCount;svcNum++)
    {
        evtInfo = TAP_GetEvent( SVC_TYPE_Tv, svcNum, &eventNum );
        if (eventNum>0)
        {
            DEBUG("UpdateEpg: deleting %d events of Service %d", eventNum, svcNum);
            for( i=eventNum-1; i>=0; i-- )
            {
                TAP_EPG_DeleteEvent( SVC_TYPE_Tv, svcNum, evtInfo[i].evtId);
                TAP_SystemProc();
            }

        }

        if (evtInfo)   // must be freed !
        {
            TAP_MemFree((unsigned char *)  evtInfo );
            evtInfo = 0;
        }
    }

    DEBUG("UpdateEpg: deleted all Epg event !");
}


Then, to fill the epg, for each single event, I use the standard api:
Code:

void *eitData;
                    // MakeEitTable from EPG_Uploader by Tony (found on TapWorld)
                    eitData = MakeEitTable( ListDvb[i].svcId /*epg.service_id*/,
                                            ListDvb[i].tsId /*epg.transport_stream_id*/,
                                            ListDvb[i].orgNetId /*epg.original_network_id*/,
                                            0 /*epg.eventid*/, startTime, duration /*epg.duration*/,
                                            runningstatus,
                                            Title /*epg.title*/, ShortDesc /*epg.shortevt*/,
                                            Desc /*epg.description*/, 0 /*epg.parental*/ );

                    if (eitData)
                    {
                        TAP_PutEvent( eitData );
                        TAP_MemFree((unsigned char *)  eitData );
                    }



Is there a better and faster way to do ?
At least for the delete events I think yes, also because I remeber to have read something (here in the forum) but with search I didn't found it !

For the put event, reading the DVB Specification I believe that it shoud possible to create a big eitData that contains all the events (then all for the 7 days...) for a specific service: that means call TAP_PutEvent only 80 times instead of near 10000 times !

Thanks, as always, for your suggestion !

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
R2-D2
Posted: Sun Nov 21, 2010 11:33 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
Yes, you can give TAP_PutEvent() a whole section's worth of data, which can be a lot of events. And you might consider disabling the normal EIT collection (via [Ed], say) if you're only going to delete them all -- that'd save all that nasty background processing, as well as the possibility of your events being overwritten by broadcast ones.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
fblasot
Posted: Mon Nov 22, 2010 4:54 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
I will try to put more events in TAP_PutEvent().

To disable EIT collection I use the TAP_ControlEit: I stop them, then delete, then fill with TAP_PutEvent and last I enable EIT Collection.

If I don't re-enable EIT collection I lost the current/next event in Info box; of course the std EIT collection will update/overwrite epg data, but it's not a big problem: if start/end time are the same it is full overwritten, sometimes I got some entry in grey...

I understood from DVB specs that current/next event are in packect, let me say, A, while epg for current service in packet B and epg for other service in packet C.
Is there a way (or a patch) to let toppy handle current/next event and ignore the full epg (both for current and other services ) ?
Or the only way is to write a new EIT process handler ?

What about a faster why to delete current epg data (or data already load by my tap ) ?
Each time I do a test of my load epg it need 5 minutes to delete previous data (the other way is to turn off/on the toppy but I don't like...) ?

Thanks...

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
bdb
Posted: Tue Nov 23, 2010 12:34 am Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
Quote:
What about a faster why to delete current epg data (or data already load by my tap ) ?

There is a function in exTap - int exTAP_EPG_DeleteAllEvents().
It has the comment 'a quick way to zap the entire firmware internal event table'.
I don't recall how quick 'quick' is, but I cant imagine ever putting such a comment by a piece of code that took several minutes to complete... but as it may make up to 14000 firmware calls, it won't be 'very quick'.

The standard tap function is very slow because it first has to search for each entry before it can delete it. This one deletes them in their stored order.

The toppy handles sections 4e/4f (now/next) differently from sections 5x/6x(schedule). Generally you want the now/next entries to prevail, as they are likely to more accurate, and may contain last minute changes such as program overruns - but it can lead to duplicate entries, especially if your schedule data does not carry the correct event_id, or originates from a different source. It is pretty simple to prune the EPG data once it is loaded, as it is all linked together by time. A scan can be run periodically to detect duplicates/overlaps and delete offending entries.

bdb
View user's profile Send private message
fblasot
Posted: Tue Nov 23, 2010 2:04 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Hi bdb,

I remeber to have seen that function in exTap when I started with trying to get DSMCC data and found exTap lib was not so updated to run with 5.13.65 fw; now it's ok, the version is 0.18 and it is up to 5.14.08, so I would try it and let you know how "quick" is this way.

Ciao.

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
fblasot
Posted: Tue Nov 23, 2010 8:48 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Hi bdb,
is right to say that you wrote exTap lib ? Tapworld says that you are the author...

I add exTap to my tap, of course I initializate the lib

Code:

...
  err = exTAP_Init();
  if (err)
  {
      DEBUG("exTap Firmware not compatible ! error %d", err);
      return 0;
  }
...


and initializate correctly.
But exTAP_EPG_DeleteAllEvents() return always -1 !!!
And looking at source program the problem should be that "fw_max_events" is still 0, even after exTap_init...

Could be the problem that my 5.13.65b is patched or becouse I use also FireBird lib ?
The order in link is exTap then FireBird.

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
R2-D2
Posted: Wed Nov 24, 2010 12:16 am Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
fblasot wrote:
Is there a way (or a patch) to let toppy handle current/next event and ignore the full epg (both for current and other services ) ?

You could try [Ed] plus EIT_Sub_(N&N).

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
bdb
Posted: Wed Nov 24, 2010 8:29 pm Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
It could well be that one of the patches interferes with the exTap firmware routine detection code.
After exTAP_Init(), these variables should all be set to non-zero values.
If any are zero, you could try just manually setting them after the exTAP_Init() call.
For 5.13.65, the expected values are (see patches.c):
Code:
   fw_max_events = 0x000036b0;
   fw_event_table = 0x80346a34;
   fw_event_table_size = 0x00000044;
   fw_event_delete = 0x801570a8;

bdb
View user's profile Send private message
fblasot
Posted: Thu Nov 25, 2010 12:43 am Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Hi bdb,
adding manually the value you suggest the exTAP_EPG_DeleteAllEvents() works fine, the full EPG is zapped in 7 seconds !!! With normal way of TAP_EPG_DeleteEvent it takes 5 minutes !
I would say that is very very fast !

I tryed to undertand why values are not correct: maybe the problem is in init_optional_fw_addresses of init.c.

After the #ifdef USE_PATCHES there are many assignment using exTAP_GetFirmwareAddress(...) and they should be executed since the -DUSE_PATCHES in makefile.
Then there is #ifdef exTAPdebug just for debug follwed by the relative #endif.
After it for, me there should be another #endif to avoid that the #else, where again there are assignment using FindInstructionPattern, are executed since exTAPdebug is not defined, and since my fw is patched some values are set to 0.

My english is not so good but I hope you have understood...

Ciao.

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
bdb
Posted: Thu Nov 25, 2010 8:44 pm Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
Unless you have recompiled exTap, I think that the precompiled exTap.a was built without USE_PATCHES (using makefile.dynamic), so the relevent line in init_optional_fw_addresses(), is:
Code:
fw_max_events = get_variable(fw_TAP_GetCurrentEvent, 0x86, 0x86);

this code looks at offset 0x86 of the firmware function, TAP_GetCurrentEvent()
if you have any patches to TAP_GetCurrentEvent(), then this would fail.
- the ifdef/endifs look ok to me...

bdb
View user's profile Send private message
fblasot
Posted: Thu Nov 25, 2010 9:15 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
After the exTAP_Init I print the value and only 'fw_event_delete' is 0, sure that depend by patched fw, for now it is not a problem I forced the values.

I review and of course you are right with #ifdef/#endif sequence...

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
fblasot
Posted: Tue Dec 07, 2010 3:24 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
I'm trying to make faster the loading of the whole epg data...

To avoid load past events I test start and end event time with toppy time but while times from epg data are in GMT the toppy time is comprensive of the offset set in setup (+ 01:00 since we are GMT+1).

Now, to let the tap works after I got the toppy time I reduce it of 1 hours...

Is there a way to know the time offset specified in setup ?
Using TAP_GetSystemVar function it seems it is not possible (and also with FireBird/exTap I don't see a specific function) ...

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message

Display posts from previous:  

All times are GMT + 1 Hour
Page 3 of 4
Goto page Previous  1, 2, 3, 4  Next

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum