Store each file as "hash/<sha1>", list of hashes as "meta/@<rev>" and serve them at "serve_file/@<r…

gce/uploader_iteration.sh

Index: gce/uploader_iteration.sh
diff --git a/gce/uploader_iteration.sh b/gce/uploader_iteration.sh
index 9003ea0cc2b7a51837e4b8e838fbfc1629e3b160..89d7014ae8efbe5793423d549a9b1220508a19ed 100755
--- a/gce/uploader_iteration.sh
+++ b/gce/uploader_iteration.sh
@@ -62,7 +62,9 @@ for REVISION in $REVISION_LIST; do
     fi
     ZIPS_DIR="$LOCAL_STORAGE_DIR/zips"
     REVS_DIR="$LOCAL_STORAGE_DIR/revs"
-    mkdir -p $ZIPS_DIR $REVS_DIR
+    META_DIR="$LOCAL_STORAGE_DIR/meta"
+    HASH_DIR="$LOCAL_STORAGE_DIR/hash"
+    mkdir -p $ZIPS_DIR $REVS_DIR $META_DIR $HASH_DIR
 
     if echo "$BLINK_REVISIONS_BLACKLIST" | grep -w $REVISION; then
         echo "Revision $REVISION is in the blacklist, skipping over"
@@ -128,6 +130,14 @@ EOF
     echo "CACHE:" >> $MANIFEST_FILE_NAME
     find -type f -not -name $MANIFEST_FILE_NAME -print | sed -e 's#^\./##' >> $MANIFEST_FILE_NAME

[mnaganov (inactive), 2015/02/24 14:40:32]

Perhaps, we can store this list in a variable and then reuse it in the new code
to avoid duplication?

[dgozman, 2015/02/25 11:37:13]

I've added manifest file there, so it's different now.

 
+    # Copy each file to hash/<sha1>, write <sha1>:<path> to meta file.
+    META_FILE_NAME="$META_DIR/@$REVISION"

[mnaganov (inactive), 2015/02/24 14:40:32]

I suggest to move this block after creating the zip file (but before popd).
Otherwise, this creates a false impression that those files also end up in the
archive.

[dgozman, 2015/02/25 11:37:14]

Done.

+    for FILE_NAME in $(find -type f -not -name $MANIFEST_FILE_NAME); do
+        FILE_SHA1=$(sha1sum $FILE_NAME | awk '{print $1}')
+        cp $FILE_NAME "$HASH_DIR/$FILE_SHA1"
+        echo "$FILE_SHA1:${FILE_NAME##./}" >> $META_FILE_NAME
+    done;
+
     zip -9r $ZIP_FILE * || exit $EXIT_ZIP_FRONTEND
     popd