[Findit] Add iterator and crash_iterator for delta test

appengine/findit/util_scripts/crash_queries/README.md

+###################### Set up enviroment variables. #####################
+##!/bin/bash
+#
+## Notice:
+## User should copy the this part to setup_enviroments.sh script and customized 
+## the path values.
+## Source this script to setup the PYTHONPATH and other environment settings for
+## pulling data from datastore App Engine Remote API. Command:

[stgao, 2016/10/13 06:38:51]

What's the entry point of the delta test? A python script or a bash shell
script?

Can we make the python script or shell script take care of this setup to avoid
manual action if possible?

[Sharu Jiang, 2016/10/13 20:31:16]

The entry point is a python script. 

Some user setups are needed since path of the appengine sdk, dir of local
checkouts and etcetera varies from user to user.

[stgao, 2016/10/14 00:55:01]

Why this can't be taken care by the python script itself? Checkout of the infra
repo has a stable dir structure.
Can these be input parameters and be part of the help description of the python
script?

[stgao, 2016/10/19 00:49:47]

And ping again

[Sharu Jiang, 2016/10/19 20:12:44]

This can be taken care by the python script itself. The reasons why I didn't do
it are:
1. For every scripts under crash_queries, we need to add a lot duplicate
sys.path.insert and they all need to be changed if the structure changes.
2. For App id and check out dir, if we make those parameters of python script,
users need to pass app_id and checkout_dir every time they run the script. To
avoid that, they needs to set default value directly in the code (which I don't
think is a good practice and may be accidentally checked in).
3. This is the practice for clusterfuzz and findit for clusterfuzz local
scripts(including delta test script). We agreed on this practice.

[stgao, 2016/10/20 01:18:37]

In that case, can we extract and share just like
appengine/findit/util_scripts/remote_api.py?
Could remote_api.py just be reused here?
Structure changes should be rare. We have been using remote_api.py for over a
year and a half, and we don't have any trouble at all.
Have you thought thoroughly on possible alternatives like below?
1. Have reasonable default values to avoid manual input all the time, but also
provide message to notify users of the default values.
2. Respect values set command line, and then in env like in .bashrc.

Why check out dir can not be derived with relative path from the python script?
I agreed there, but not here :)
As we are redesigning, we could rethink.

[Sharu Jiang, 2016/10/20 20:38:20]

Yes, those findit_dir, third_party_dir, appengine_dir inserts can be shared,
though util_script_dir insert cannot be avoided for every script, but I think
this little amount of duplication is acceptable.
I mentioned "To avoid that, they needs to set default value directly in the
code". Ok, I can set default values. However the only concern remains right now
we are testing in findit-for-me-dev, if we defaults to that, later we have to
directly change code to change the default value.
Yes, we can, however we'd better put it in the home directory to avoid checking
into repo.

[stgao, 2016/10/21 01:36:34]

We could check environment setting as in my last comment for such values before
using the default values.

[Sharu Jiang, 2016/10/21 18:29:40]

Done.

+##   source setup_enviroments.sh
+#
+## This must be set to the directory in which Findit source code is
+## checked out, eg: ~/chrome-internal if the checkout is in
+## ~/chrome-internal/findit.
+#export FINDIT_ROOT_DIR='root/path/to/findit'
+#
+## Since findit doesn't want to cover the tests of
+#export UTIL_SCRIPTS_DIR=$FINDIT_ROOT_DIR/util_scripts
+#
+##This must be set to the third_party directory under findit.
+#export THIRD_PARTY_DIR=${FINDIT_ROOT_DIR}/third_party
+#
+## This must be set to your appengine SDK directory.
+#APPENGINE_SDK_DIR='path/to/google_appengine'
+#
+#if [[ $PYTHONPATH != *$APPENGINE_SDK_DIR* ]]
+#then
+#  export PYTHONPATH=$UTIL_SCRIPTS_DIR:$FINDIT_ROOT_DIR:$THIRD_PARTY_DIR:$APPENGINE_SDK_DIR
+#fi
+#
+## This should be set to your app id.
+#export APP_ID="your app id"
+#
+## This must be set to the directory to put local checkout in.
+#export CHECKOUT_ROOT_DIR='path/to/checkout/root/path'