For discussions about development of TAPs, patches and other software for the Toppy

Moderator: Technical

R2-D2
Frequent contributor
Posts: 12148
Joined: Mon Dec 18, 2006 11:15 am
Contact:

Simple and fast .REC conversion

Post by R2-D2 »

Hi,

I've been looking at the Toppy's .rec format and I've found that all I need to do to turn a .rec file into something that is generally usable is to remove the (roughly) 4KB header (to the first 0x47 sync byte). This leaves a normal MPEG Transport Stream (.ts) file which lots of things can play and manipulate.

However, it's desperately inefficient to chop the top off a file since (on most filesystems I've been exposed to) you end up having to basically copy the whole file. So it struck me that a much better idea might be to rewrite the 4KB header part. So I tried inserting 0x47 sync bytes starting from the beginning and at 188 byte intervals (as if it were part of a TS file). This worked perfectly and lined up exactly with the first real 0x47 sync byte. And the file then played perfectly in Media Player Classic (which it didn't if just one of those 0x47 sync bytes was missing).

Now, after the last inserted 0x47 sync byte in the header is invariably a big chunk of zero bytes, which I reckon could be used to store the bytes that had been changed so you could actually change the file back to the original Toppy .rec format just as easily. In fact, in the examples I've looked at there was only one byte that was changed that was variable (i.e. part of the text of the recording description) and couldn't otherwise be easily replaced.

So, it strikes me that it would be fairly easy to write a TAP and/or computer (PC/Mac/Linux) program to do this update (and maybe change the file extension at the same time to .TS to signal the change).

Silly idea or not?
nwhitfield
Site Admin
Posts: 9589
Joined: Sun Mar 20, 2005 10:11 pm
Contact:

Post by nwhitfield »

Hmmm; that sounds terribly interesting. Did you just use a text editor, or knock up a tool to do the job?

I shall have to try that with some of the .rec files sitting on the NAS and see if the media player is more amenable to looking at them after that.

Nigel.
Support this site - make a donation to our running costs
R2-D2
Frequent contributor
Posts: 12148
Joined: Mon Dec 18, 2006 11:15 am
Contact:

Post by R2-D2 »

nwhitfield wrote:Hmmm; that sounds terribly interesting. Did you just use a text editor, or knock up a tool to do the job?
I used a hex editor called 010Editor, which has the advantage that I could set the width to 188 bytes and have all the sync bytes line up in the first column.
bdb
Frequent contributor
Posts: 499
Joined: Tue Oct 18, 2005 10:50 pm

Post by bdb »

Any 'good' codec should be able to cope with an errored bitstream - which is effectively what the .rec header appears as. Sadly, many do get fooled.

rec2ts_v0.02.zip is pc application that will remove the .rec header for you, and create a valid .ts file. It then decodes the .rec header and puts in in a spreadsheet.

Overwriting the .rec header rather than rewriting the entire file would certainly speed up the process; but it doesn't take that long to rewrite a 4GB file on a PC.
If I get a chance, I will add an option to rec2ts to offer this choice.

bdb
iron
Regular contributor
Posts: 91
Joined: Fri Jun 03, 2005 1:01 pm
Location: Tunbridge Wells, UK

Post by iron »

I would overwrite the rec header with transport stream null packets, i.e. packets with a PID of 0x1FFF. The null packets are used for padding in the transmitted stream and decoders and editors are designed to ignore them.

iron.
Model: TF5800PVR - Silver Firmware: Recommended_5800; TAPS: UK Timer 1.2, USB Accelerator 1.0, eit_sub, seccacheUK; cables: 10m TF to PC only, 2 metres both ways!
R2-D2
Frequent contributor
Posts: 12148
Joined: Mon Dec 18, 2006 11:15 am
Contact:

Post by R2-D2 »

iron wrote:I would overwrite the rec header with transport stream null packets, i.e. packets with a PID of 0x1FFF.
Ah yes, that would be better. So three bytes per header packet, meaning 60 bytes in all. They still fit in the (as far as I've ever seen) unused last part of the header. I don't know if this is the best place to post this, or if it would be useful to anyone else, but I've converted a quick program I'd written to PHP (for increased portability). This does the quick translation of .REC files to .TS files (including renaming them) -- and back again, too!

Code: Select all

<?php
/* rec2ts.php - Convert Toppy .rec to .ts &#40;and back again&#41; */

/* Rename converted file? */
$dorename = TRUE;

if&#40;count&#40;$argv&#41; > 1&#41;
  $filename = $argv&#91;1&#93;;
else
  exit&#40;"No filename supplied\n"&#41;;

$handle = @fopen&#40;$filename, 'r+b'&#41;;
if&#40;$handle === FALSE&#41;
  exit&#40;"Failed to open $filename\n"&#41;;

fseek&#40;$handle,0&#41;;
$hdr = fread&#40;$handle, 3760&#41;;
if&#40;$hdr === FALSE&#41;
  exit&#40;"Error reading from $filename\n"&#41;;

$arr = str_split&#40;$hdr,188&#41;;
if&#40;$arr&#91;0&#93;&#91;0&#93;==chr&#40;0x47&#41;&#41; &#123;
  foreach&#40;$arr as $key => $_&#41; &#123;
    for&#40;$i=0; $i < 3; $i++&#41;
      $arr&#91;$key&#93;&#91;$i&#93; = $arr&#91;19&#93;&#91;$key * 3 + 3 + $i&#93;;
  &#125;
  echo "Reverted to Topfield rec file\n";
  $newname = preg_replace&#40;'/.ts$/i','.rec',$filename&#41;;
&#125; else &#123;
  $repl = array&#40;chr&#40;0x47&#41;,chr&#40;0x1F&#41;,chr&#40;0xFF&#41;&#41;;
  foreach&#40;$arr as $key => $_&#41; &#123;
    for&#40;$i=0; $i < 3; $i++&#41; &#123;
      $arr&#91;19&#93;&#91;$key * 3 + 3 + $i&#93; = $arr&#91;$key&#93;&#91;$i&#93;;
      $arr&#91;$key&#93;&#91;$i&#93; = $repl&#91;$i&#93;;
    &#125;
  &#125;
  echo "Converted to TS file\n";
  $newname = preg_replace&#40;'/.rec$/i','.ts',$filename&#41;;
&#125;

// Write the string to the file
fseek&#40;$handle,0&#41;;
if&#40;fwrite&#40;$handle, implode&#40;"",$arr&#41;&#41; === FALSE&#41;
  exit&#40;"Error writing to $filename\n"&#41;;

fclose&#40;$handle&#41;;

// Rename the file if needed
if&#40;$dorename && $newname !== $filename&#41;
  rename&#40;$filename,$newname&#41;;

/* End */
?>
matt
Forum moderator
Posts: 2305
Joined: Wed Mar 30, 2005 6:59 pm
Location: St Ives, Cornwall Transmitter: Redruth
Contact:

Post by matt »

I've converted the above script to a droplet for Mac OS X.

Drag and drop a REC or TS file onto it to do a quick conversion back and forth.

notes: only supports one file at a time. doesn't have a fancy icon.

download: http://www.gingerbeardman.com/toppy-rec2ts.zip (58kb)

if you find it useful please let me know.

matt
PVR: TF5800, MyStuff recommended firmware, new PSU, 250GB Spinpoint HDD, Zalman ZM-OP1 fan TAPs: QuickJump 1.72, MyStuff 6.4, EPG2MEI 0.96, Extend 1.7, Bookmark 3, UK Auto Scheduler 0.73, SecCache 0.4, EIT Sub 0.6, SDS 1.3, TAP Commander 1.34, Font Manager 1.0d. Others: HDFW 2.3a, Channel Manager 5.1, FastScanGUI, MyInfo TV: Philips 37PFL9603D
richie
Frequent contributor
Posts: 156
Joined: Wed Mar 30, 2005 1:05 pm
Location: Ash Vale (Hannington)

Post by richie »

Out of interest, has anyone ported this into tap format, for use with recordings that a user knows are only to be transferred off the toppy? ie It gets the rec files ready as ts files.

I ask because my XP machine is very short of disk space and rec2ts on win makes a (large) copy of the file first before deleting the original!
R2-D2
Frequent contributor
Posts: 12148
Joined: Mon Dec 18, 2006 11:15 am
Contact:

Post by R2-D2 »

richie wrote:Out of interest, has anyone ported this into tap format
There's little point in doing it on the Toppy since the .ts is not playable. To use the script on a PC all you need is PHP (the .exe and .dll as a minimal installation).
DB1
Frequent contributor
Posts: 728
Joined: Wed Mar 30, 2005 6:45 pm
Location: Orpington

Post by DB1 »

Err, why don't you use a player that understands .rec? Like Mplayer.
TF5800, F/W: 5.13.65AbBfBqC0CbCeCkCwCyDEcEeEfErEsEvEzFFsGmHHeIKtNfOtPPcPePsRRaRhRpRsSSdSrStT2TdTfTpUUuXXpXwXl TAPs: TF5000 Display v1.53; QuickJump 1.72; Power Manager v2.2; MyInfo B5.6; DescriptionExtender 2.23; Remote Extender 1.6; Archivev1.0a; mei2archive BETA 3.8l7; EPGnavigator v5.1c; UK Auto Scheduler v0.73.1; Extend v1.7; Power Restore V0.7.8
Tx: CP
Sig generated by MyInfo on 14/4/13
richie
Frequent contributor
Posts: 156
Joined: Wed Mar 30, 2005 1:05 pm
Location: Ash Vale (Hannington)

Post by richie »

It's for conversion into divx for my portable media player. I do stream using vlc when I want to watch stuff on my pc.
richie
Frequent contributor
Posts: 156
Joined: Wed Mar 30, 2005 1:05 pm
Location: Ash Vale (Hannington)

Post by richie »

So to clarify, for a 2Gb rec file, I need 4gb space.

Because rec2ts creates a copy before deleting the original, before something then converts it from ts into mpg. Though I have in the past used mencoder, which can convert it from rec into divx, I am now playing with mp4 scripts on my asus, and the only one i can find available as an ipkg is ffmpeg. ffmpeg is also faster than mencoder.

Whereas if there was a tap which could convert a file I knew I wanted to be on avi only, ie never watch on my toppy, then I would only need about 2.3 Gb on my pc - 2 Gb fo the ts plus 300mb or so for the avi being created.

This is a very custom application!

Perhaps I should just buy a new bigger laptop drive!
R2-D2
Frequent contributor
Posts: 12148
Joined: Mon Dec 18, 2006 11:15 am
Contact:

Post by R2-D2 »

richie wrote:the only one i can find available as an ipkg is ffmpeg.
I expect you can run PHP on your device.
captain747480
Frequent contributor
Posts: 499
Joined: Sat Jun 25, 2005 9:26 am
Location: London Transmitter: Crystal Palace

Post by captain747480 »

R2-D2 wrote:
iron wrote:I would overwrite the rec header with transport stream null packets, i.e. packets with a PID of 0x1FFF.
Ah yes, that would be better. So three bytes per header packet, meaning 60 bytes in all. They still fit in the (as far as I've ever seen) unused last part of the header. I don't know if this is the best place to post this, or if it would be useful to anyone else, but I've converted a quick program I'd written to PHP (for increased portability). This does the quick translation of .REC files to .TS files (including renaming them) -- and back again, too!

Code: Select all

<?php
/* rec2ts.php - Convert Toppy .rec to .ts &#40;and back again&#41; */

/* Rename converted file? */
$dorename = TRUE;

if&#40;count&#40;$argv&#41; > 1&#41;
  $filename = $argv&#91;1&#93;;
else
  exit&#40;"No filename supplied\n"&#41;;

$handle = @fopen&#40;$filename, 'r+b'&#41;;
if&#40;$handle === FALSE&#41;
  exit&#40;"Failed to open $filename\n"&#41;;

fseek&#40;$handle,0&#41;;
$hdr = fread&#40;$handle, 3760&#41;;
if&#40;$hdr === FALSE&#41;
  exit&#40;"Error reading from $filename\n"&#41;;

$arr = str_split&#40;$hdr,188&#41;;
if&#40;$arr&#91;0&#93;&#91;0&#93;==chr&#40;0x47&#41;&#41; &#123;
  foreach&#40;$arr as $key => $_&#41; &#123;
    for&#40;$i=0; $i < 3; $i++&#41;
      $arr&#91;$key&#93;&#91;$i&#93; = $arr&#91;19&#93;&#91;$key * 3 + 3 + $i&#93;;
  &#125;
  echo "Reverted to Topfield rec file\n";
  $newname = preg_replace&#40;'/.ts$/i','.rec',$filename&#41;;
&#125; else &#123;
  $repl = array&#40;chr&#40;0x47&#41;,chr&#40;0x1F&#41;,chr&#40;0xFF&#41;&#41;;
  foreach&#40;$arr as $key => $_&#41; &#123;
    for&#40;$i=0; $i < 3; $i++&#41; &#123;
      $arr&#91;19&#93;&#91;$key * 3 + 3 + $i&#93; = $arr&#91;$key&#93;&#91;$i&#93;;
      $arr&#91;$key&#93;&#91;$i&#93; = $repl&#91;$i&#93;;
    &#125;
  &#125;
  echo "Converted to TS file\n";
  $newname = preg_replace&#40;'/.rec$/i','.ts',$filename&#41;;
&#125;

// Write the string to the file
fseek&#40;$handle,0&#41;;
if&#40;fwrite&#40;$handle, implode&#40;"",$arr&#41;&#41; === FALSE&#41;
  exit&#40;"Error writing to $filename\n"&#41;;

fclose&#40;$handle&#41;;

// Rename the file if needed
if&#40;$dorename && $newname !== $filename&#41;
  rename&#40;$filename,$newname&#41;;

/* End */
?>
What's the syntax for running this?

I've tried

/opt/bin/php rec2ts.php filename.rec

but I get 'Failed to open filename.rec'
TF5800 HD103SI 1Tb HDD, TS On,F/W: MS6 Recommended F/W 12/9/2009 -Sy+Pe
TAPs: MyStuff 6.4; TF5000 Display v1.53; MHEG Control A2f; EPG2MEI v0.96; TAP Commander 1.0.3; MyInfo B5.5; AccurateBM v0.3; SecCache (UK) v0.4; Font Manager 1.0d
Other Kit:LG 37LE5900 TV, Zotac ND-22 HTPC running XMBC, Sky+ HD, Panasonic HT-520 Home Cinema, Harmony 885, Asus WL-500GX with Oleg's, 2Tb NAS (Mirrored), Flirc USB Keyboard Emulator
R2-D2
Frequent contributor
Posts: 12148
Joined: Mon Dec 18, 2006 11:15 am
Contact:

Post by R2-D2 »

captain747480 wrote:I've tried

/opt/bin/php rec2ts.php filename.rec
That looks good to me, although is your recording really called "filename.rec"? And there are no paths mentioned, except on the php command, so is it actually in the same directory as the script?
Post Reply