newznab-tmux/misc/testing/spotnab.php

<?php
// Author l2g
// Date: Mar 17th, 2013
// Version: 0.97.4
//
// Description:
// This script is an implementation of spotnet and newznab to which
// multiple servers can share comments and information amongst each other
//
// The implementation is based loosely on how spotweb works. If you don't know
// what spotweb is, feel free to Google it.
//
// With an spotnab, you have 2 options:
//	1. Fetch comments (and potential other things as we dream them up) from
//		other newznab sources and apply them to your own comment section.
//
//		Fetched content is scanned and decrypted using a password that the
//		newznab server who posted it chooses to share with you.  If you can't
//		decrypt the post then you move along.  Since all newznab servers choose
//		to share and or not to share.  Spotnab will only populate your database
//		with comments from the sources who choose to share their comments
//		with you.
//
//		A few public keys will be configured out of the box so that you can
//		retrieve your comments from existing active sites should you choose
//		to enable them.
//
//	2. Post comments;  send all the comments made by your local users of your
//		site to usenet encrypted by your own secure private key. Only those you
//		share your public key will be allowed to decrypt it for their own
//		server.
//
// Releases are uniquely bundled and shared across NewzNab servers through the
// use of the releases table. To uniquely identify a release across remote
// servers we need to extract the Message-ID from the first segment of the first
// file of the matched release (as it was posted to usenet). This ID is unique
// to usenet, so it'll be unique to us as well. We can achive this information
// from parsing the NZB file generated by the NewzNab server.
//
// This value can be certain to not conflict and uniquely identify a release
// across all NewzNab Servers.
//
// Content is posted and retrieved to Usnet in JSON encrypted using the key
// belonging to the NewzNab Server.
//
// Structure is as follows:
//   {
//     server: {
//         code: <string>,
//         title: <string>,
//     },
//     postdate_utc: 'YYYY-MM-DD hh:mm:ss.ms',
//     comments: [
//       {
//         gid: <char[32]>,
//         cid: <char[32]>,
//         comment: <string>,
//         is_visible: <bool>,
//         username: <string>,
//         postdate_utc: 'YYYY-MM-DD hh:mm:ss',
//       }
//     ]
//   }
//
//	server:          Meta information on the server that posted the spot.
//
//	server
//   -/ code:        This is taken from the users local database `site` table.  It is
//                    the `code` the user configured their newznab server as.
//
//   -/ title:       This is also taken from the users local database `site` table. It
//                      is the the `title` the user configured their newznab server as.
//
//	postdate_utc:    This is the current time measured in the number of seconds since
//                    the Unix Epoch (January 1 1970 00:00:00 GMT). This represents
//                    the local UTC time of when this message was posted. It doesn't
//                    reflect the age of the content to follow.
//
//	comments:        This is an array of comment objects
//
//	comments
//   -/ gid:         This is the Global Identifier for the release the comment
//                    is in relation to.
//
//   -/ cid:         This is a unique global identifier used to identify the comment
//                    itself.  This allows the server to publish the comment again
//                    if there is an update to it's data (such as toggling it to a
//                    hidden type).  This should be used to prevent a duplicate
//                    entry in the comments table when dealing with remote content.
//
//   -/ comment:     This is the actual comment itself associated with the release
//
//   -/ is_visible:  Whether or not the comment should be marked visible or not.
//
//   -/ username:    The username of the person who made the comment.
//
//   -/ postdate_utc: The timestamp (in UTC) that the comment was created at.
//

/*

INSTRUCTIONS
============

Both posting and fetching will 'NOT' work unless you've generated
Global Release ID values for all of your current releases... this is
done by running the command:

	#> php spotnab.php -g

POSTING
	1. To run all of the SQL statements identifed above

	2. You 'must' have all Global Release ID's defined for each
		release or the comments associated with it will not be
		included in the post.

	3. Generate yourself your own set of SSL keys:
		#> php spotnab.php -k

	4. At this point your ready to post... If you run the following SQL
		statement, you'll mark all your existing comments:

		-- Ensure spotnab posting is enabled:
		UPDATE `site` set `value` = '1' WHERE `setting` = 'spotnabpost';

		-- Posting actually uses this date as a reference to when
		-- it last posted.  So set it to a date so far back in time
		-- that all your existing comments become declared as unposted.

		UPDATE `site` set `updateddate` = '1980-01-01 00:00:00'
			WHERE `setting` = 'spotnabpost' AND `value` = '1';

FETCHING
	1. You need to acquire and enable sources.  A source is as simple
		as someone providing you their public key they generated using
		the --keygen (-k) switch.

	2. Fetch releases:
		#> php spotnab.php -f

	3. Consider fetching your own posts back just to test things out...:

			SET @key := (SELECT value FROM settings WHERE setting = 'spotnabsitepubkey');
			INSERT INTO  `spotnabsources` (`ID` , `username`, `useremail` ,
				`usenetgroup`, `publickey` , `active` ,
				`description` , `lastupdate` )
			VALUES ( NULL , 'nntp', 'spot@nntp.com',  'alt.binaries.backup', @key,
				'1',  '1',  'My Key Fetch Test', NULL);

		Then you can run the fetch release command again to poll for
		stuff you posted already (if you did).

			#> php spotnab.php -f

SHARING
	1. Use the -k switch on the tool to retrieve your public key...
		share it... let people use the comments posted on your site
		get them to do the same to you.  Eliminating crappy content
		couldn't be more easy.

		Spotnab now has a built in feature to post a discovery message
		for others to grab automatically. Just call:
			#> php spotnab.php -b

		You can also fetch messages from those who have already
		posted their own discovery message by calling:
			#> php spotnab.php -d

		Just note that anything auto-discovered is disabled by default
		so you need to go into your options and 'enable' it if you
		recognize or want to test out the source.
*/

require_once dirname(__FILE__) . '/../../www/config.php';

// Subject Lines will always be a hash key that helps scanning identify whether
// or not the contents are valid or not.  the hash key contains is built
// by applying the password key against the md5 some of the message content
// itself.  Unmatched content is ignored.

$shortopts = "";
$shortopts .= "G";
$shortopts .= "g";
$shortopts .= "r";
$shortopts .= "p";
$shortopts .= "f";
$shortopts .= "t";
$shortopts .= "k";
$shortopts .= "o";
$shortopts .= "K";
$shortopts .= "d";
$shortopts .= "b";
$shortopts .= "F::";

$longopts = array(
	"post",
	"fetch",
	"fetch-backfill::",
	"discover",
	"broadcast",
	"test",
	"populate-gid",
	"populate-fix-gid",
	"soft-reset",
	"keygen",
	"force-keygen",
	"clean-orphan-comments"
);

$options = getopt($shortopts, $longopts);

function display_help(){
	echo "\n";
	echo "SpotNab v0.97.4, Author: l2g\n";
	echo "Syntax: spotnab.php <action>\n";
	echo "\n";
	echo "Actions:\n";
	echo "   -g, --populate-gid        This could be considered phase one of "
			."this project.\n";
	echo "                              requiring that your releases database"
			." table is up to date\n";
	echo "                              with all GID (Global Identifiers) so"
			." it can correctly\n";
	echo "                              communicate with other servers that "
			."share the same content\n";
	echo "                              from the servers configured.\n";
	echo "   -G, --populate-fix-gid    Same as -g except broken nzb files are "
			."also broken releases.\n";
	echo "                              Specifying this switch will remove "
			."these dead releses.\n";
	echo "   -f, --fetch               Get latest spotnab comments from "
			."usenet using the information\n";
	echo "\n";
	echo "   -F=DAYS\n";
	echo "   --fetch-backfill=DAYS     Get latest spotnab comments as far back"
			." as the days specified.\n";
	echo "\n";
	echo "   -p, --post                Post latest updates from local system "
			."to usenet.\n";
	echo "\n";
	echo "   -k, --keygen              Generate a new SSL Public/Private Key "
			."pair only if one isn't\n";
	echo "                              already generated.\n";
	echo "   -K, --force-keygen        Generate a new SSL Public/Private Key "
			."pair\n";
	echo "\n";
	echo "   -d, --discover            Attempt to discovery all sources available.\n";
	echo "\n";
	echo "   -o, --clean-orphan-comments\n";
	echo "                             Eliminate all fetched comments that you do not have a"
			." release\n";
	echo "                             associated with.\n";
	echo "\n";
	echo "   -b, --broadcast           Broadast information for others so they can discover.\n";
	echo "\n";
	echo "   -r, --soft-reset          Safely resets sources as though they were\n";
	echo "                              just added. This is ideal to do if you\n";
	echo "                              change usenet servers.\n";
	echo "   -t, --test                Produces a whole lot of garbage, but "
			."is used for testing\n";
	echo "                              the internals of the class...\n";
	echo "\n";
}

if(!$options){ display_help(); exit(1);}
if(!count($options)){ display_help(); exit(1);}

$delete_broken_releases = false;
if(array_key_exists("G", $options) ||
	array_key_exists("populate-fix-gid", $options)){
	echo "Updating GID in releases table + fix ...";
	$spotnab = new SpotNab();
	$spotnab->processGID(0,5000,true);
	echo "Done\n";
}

if(array_key_exists("g", $options) ||
	array_key_exists("populate-gid", $options)){
	echo "Updating GID in releases table ...";
	$spotnab = new SpotNab();
	$spotnab->processGID();
	echo "Done\n";
}

if(array_key_exists("r", $options) ||
	array_key_exists("soft-reset", $options)){
	echo "Soft Reseting Spotnab... ";
	$spotnab = new SpotNab();
	$spotnab->soft_reset();
	echo "Done\n";
}

$force_keygen_save = false;
if(array_key_exists("K", $options) ||
	array_key_exists("force-keygen", $options)){
	$spotnab = new SpotNab();
	$spotnab->keygen(true, true);
}
else if(array_key_exists("k", $options) ||
	array_key_exists("keygen", $options)){
	$spotnab = new SpotNab();
	$spotnab->keygen(true);
}

if(array_key_exists("p", $options) ||
	array_key_exists("post", $options)){
	echo "Posting... ";
	$spotnab = new SpotNab();
	$spotnab->post();
	echo "Done\n";
}

if(array_key_exists("d", $options) ||
	array_key_exists("discover", $options)){
	echo "Discovering... ";
	$spotnab = new SpotNab();
	$spotnab->fetch_discovery();
	echo "Done\n";
}

if(array_key_exists("F", $options) ||
	array_key_exists("fetch-backfill", $options)){
	$days = array_key_exists("F", $options)?$options["F"]:$options["fetch-backfill"];
    try{
    	$days = abs(intval($days));
    }catch(Exception $e){
		$days = -1;
    }

    if($days <= 0){
    	echo "Error: A SpotNab fetch backfill requires you specify the number of days to look back.\n";
    	echo "Syntax: php spontnab.php -F=<days>\n";
        exit(1);
    }
	echo "Fetching $days day(s) back ... ";
	$spotnab = new SpotNab();
	$spotnab->fetch(time()-($days*86400));
	echo "Done\n";


}else if (array_key_exists("f", $options) ||
	array_key_exists("fetch", $options)){
	echo "Fetching... ";
	$spotnab = new SpotNab();
	$spotnab->fetch();
	echo "Done\n";
}

if(array_key_exists("b", $options) ||
	array_key_exists("broadcast", $options)){
	echo "Broadcasting... ";
	$spotnab = new SpotNab();
	$spotnab->post_discovery();
	echo "Done\n";
}

if(array_key_exists("t", $options) ||
	array_key_exists("test", $options)){
	$spotnab = new SpotNab();

	if($spotnab->has_openssl())
	{
		printf("%s INFO - Testing SSL Key Generator ...",
					date("Y-m-d H:i:s"));
		$keys = $spotnab->keygen(false);
		if(is_array($keys) &&
			array_key_exists("pubkey", $keys) &&
			array_key_exists("prvkey", $keys))
		{
			$prvkey = $spotnab->decompstr($keys['prvkey']);
			$pubkey = $spotnab->decompstr($keys['pubkey']);
			$refc = $spotnab->getRandomStr(80);
			$refd = $spotnab->decrypt($spotnab->encrypt($refc, $prvkey), $pubkey);
			echo ($refc == $refd)?"Successful!\n":"Failed!\n";
		}else{
			echo "Failed!\n";
		}

		printf("%s INFO - Testing SSL encryption/decryption ...",
				date("Y-m-d H:i:s"));
		$preMsg = $spotnab->getRandomStr(800);
		$postMsg = $spotnab->decrypt($spotnab->encrypt($preMsg));
		if($postMsg === false){
			echo "Failed!\n";
		}else if(!strcmp($preMsg, $postMsg)){
				echo "Successful!\n";
		}else{
			echo "Failed!\n";
		}

		printf("%s INFO - Testing small message encode/decode ...",
					date("Y-m-d H:i:s"));
		$before = array(
			'server' => array(
				'code' => 'l2g',
				'title' => 'l2g newznab'
			),
			'postdate_utc' => $spotnab->local2utc(),
			'comments' => array(
				array(
					'gid' => 'ABCDEFHIJKLMNOPQRSTUVWXYZ0123456',
					'cid' => 'ABCDEFHIJKLMNOPQRSTUVWXYZ0123456',
					'comment' => 'testing comment 1',
					'username' => 'l2g',
					'is_visible' => 1,
					'postdate_utc' => $spotnab->local2utc(time()-86400)
				),
				array(
					'gid' => 'ABCDEFHIJKLMNOPQRSTUVWXYZ0123456',
					'cid' => 'ABCDEFHIJKLMNOPQRSTUVWXYZ0123456',
					'username' => 'l2g-hater',
					'is_visible' => 1,
					'comment' => 'testing comment 2',
					'postdate_utc' => $spotnab->local2utc(time()-86000)
				)
			)
		);
		$article = $spotnab->encodePost($before, Null, true);
		if($article !== false){
			$after = $spotnab->decodePost($article[1]);
			if($before === $after){
				echo "Successful!\n";
			}else{
				echo "Failed!\n";
			}
		}else{
			echo "Failed!\n";
		}
		printf("%s INFO - Testing big message encode/decode ...",
					date("Y-m-d H:i:s"));
		$before = array(
			'server' => array(
				'code' => 'l2g',
				'title' => 'l2g newznab'
			),
			'postdate_utc' => $spotnab->local2utc(),
			'comments' => []
		);
		for($i=0;$i<3000;$i++){
			// Build large post
			$before['comments'][] =array(
				'gid' => 'ABCDEFHIJKLMNOPQRSTUVWXYZ0123456',
				'cid' => 'ABCDEFHIJKLMNOPQRSTUVWXYZ0123456',
				'comment' => $refc = $spotnab->getRandomStr(rand(15,200)),
				'username' => 'bb',
				'is_visible' => 1,
				'postdate_utc' => $spotnab->local2utc(time()-86400)
			);
		}
		$article = $spotnab->encodePost($before, Null, true);
		if($article !== false){
			$after = $spotnab->decodePost($article[1]);
			if($before === $after){
				echo "Successful!\n";
			}else{
				echo "Failed!\n";
			}
		}else{
			echo "Failed!\n";
		}

		printf("%s INFO - Testing fake usenet parse ...",
				date("Y-m-d H:i:s"));
		// Fake group hash table
		$hash = array(array(
			'ID' => 0,
			'key' => $spotnab->decompstr($keys['pubkey']),
			'user' => 'nntp',
			'email' => 'spot@nntp.com',
			// We want to find new content, so to make our header
			// new, we need to take our ref time and back down
			// one second so it can be processed..
			'ref' => $article[2]['Epoch']-1
		));

		// Fake headers (use debug information from encodePost)
		$headers = array($article[2]);

		$matched = $spotnab->process_comment_headers($headers, $hash, false);
		if($matched !== false){
			$inserted = $matched[0];
			$updated = $matched[1];
			echo ($matched > 0)?"Successful!\n":"Failed!\n";
		}else{
			echo "Failed\n";
		}
	}else{
		printf("%s WARNING - openssl is not correctly installed; broadcasts and posts will be disabled.\n",
				date("Y-m-d H:i:s"));
	}

	printf("%s INFO - Testing UTC/Local conversions [1/6]...",
				date("Y-m-d H:i:s"));
	$refa = $spotnab->utc2local();
	$refb = $spotnab->utc2local($spotnab->local2utc($refa));
	echo ($refa == $refb)?"Successful!\n":"Failed!\n";
	printf("%s INFO - Testing UTC/Local conversions [2/6]...",
				date("Y-m-d H:i:s"));
	$refa = $spotnab->local2utc();
	$refb = $spotnab->local2utc($spotnab->utc2local($refa));
	echo ($refa == $refb)?"Successful!\n":"Failed!\n";
	printf("%s INFO - Testing UTC/Local conversions [3/6]...",
				date("Y-m-d H:i:s"));
	$refa = $spotnab->local2utc(date("Y-m-d H:i:s"));
	$refb = $spotnab->local2utc($spotnab->utc2local($refa));
	echo ($refa == $refb)?"Successful!\n":"Failed!\n";
	printf("%s INFO - Testing UTC/Local conversions [4/6]...",
				date("Y-m-d H:i:s"));
	$refa = $spotnab->utc2local(time());
	$refb = $spotnab->utc2local($spotnab->local2utc($refa));
	echo ($refa == $refb)?"Successful!\n":"Failed!\n";
	printf("%s INFO - Testing UTC/Local conversions [5/6]...",
				date("Y-m-d H:i:s"));
	$refa = $spotnab->local2utc(time());
	$refb = $spotnab->local2utc($spotnab->utc2local($refa));
	echo ($refa == $refb)?"Successful!\n":"Failed!\n";
	printf("%s INFO - Testing UTC/Local conversions [6/6]...",
				date("Y-m-d H:i:s"));
	$refa = $spotnab->utc2local(gmdate("Y-m-d H:i:s"));
	$refb = $spotnab->utc2local($spotnab->local2utc($refa));
	echo ($refa == $refb)?"Successful!\n":"Failed!\n";
}

if(array_key_exists("o", $options) ||
	array_key_exists("clean-orphan-comments", $options)){
	echo "Removing orphan comments...";
	$spotnab = new SpotNab();
	printf("%d record(s) removed.\n", $spotnab->orphan_comment_clean());
}