[telemetry] Add Python dynamic analysis find_dependencies.

[telemetry] Add Python dynamic analysis find_dependencies.

If we import all the modules in Telemetry, then our sys.modules should contain most of Telemetry's dependencies. This check is fairly fast, so we can do it on all files, in addition to using modulefinder on a few select files.


BUG=
TEST=Diif the output of src/tools/telemetry/build/generate_telemetry_harness.sh with and without this patch. Ensure it makes sense. (The new entries are all py_trace_event.)

[nednguyen, 2015-07-28 00:23:32]

nednguyen@google.com changed reviewers:
+ nednguyen@google.com

[nednguyen, 2015-07-28 00:23:33]

https://codereview.chromium.org/1254173003/diff/1/tools/telemetry/telemetry/i...
File tools/telemetry/telemetry/internal/util/find_dependencies.py (right):

https://codereview.chromium.org/1254173003/diff/1/tools/telemetry/telemetry/i...
tools/telemetry/telemetry/internal/util/find_dependencies.py:53: module_path =
root + '.py'
We have issues with stale *pyc file. Probably worth checking whether .py file
exist?

[dtu, 2015-07-28 00:40:49]

https://codereview.chromium.org/1254173003/diff/1/tools/telemetry/telemetry/i...
File tools/telemetry/telemetry/internal/util/find_dependencies.py (right):

https://codereview.chromium.org/1254173003/diff/1/tools/telemetry/telemetry/i...
tools/telemetry/telemetry/internal/util/find_dependencies.py:53: module_path =
root + '.py'
On 2015/07/28 00:23:33, nednguyen wrote:
> We have issues with stale *pyc file. Probably worth checking whether .py file
> exist?

Done. Good catch. I had run remove_stale_pyc_files.py before testing this patch,
but forgot to add the check in.

[nednguyen, 2015-07-28 01:40:37]

On 2015/07/28 00:40:49, dtu wrote:
>
https://codereview.chromium.org/1254173003/diff/1/tools/telemetry/telemetry/i...
> File tools/telemetry/telemetry/internal/util/find_dependencies.py (right):
> 
>
https://codereview.chromium.org/1254173003/diff/1/tools/telemetry/telemetry/i...
> tools/telemetry/telemetry/internal/util/find_dependencies.py:53: module_path =
> root + '.py'
> On 2015/07/28 00:23:33, nednguyen wrote:
> > We have issues with stale *pyc file. Probably worth checking whether .py
file
> > exist?
> 
> Done. Good catch. I had run remove_stale_pyc_files.py before testing this
patch,
> but forgot to add the check in.

Since this no longer block the other patch. I think we can close this issue for
now as we are still unsure about the best approach?

[dtu, 2015-07-30 08:23:05]

On 2015/07/28 01:40:37, nednguyen wrote:
> On 2015/07/28 00:40:49, dtu wrote:
> >
>
https://codereview.chromium.org/1254173003/diff/1/tools/telemetry/telemetry/i...
> > File tools/telemetry/telemetry/internal/util/find_dependencies.py (right):
> > 
> >
>
https://codereview.chromium.org/1254173003/diff/1/tools/telemetry/telemetry/i...
> > tools/telemetry/telemetry/internal/util/find_dependencies.py:53: module_path
=
> > root + '.py'
> > On 2015/07/28 00:23:33, nednguyen wrote:
> > > We have issues with stale *pyc file. Probably worth checking whether .py
> file
> > > exist?
> > 
> > Done. Good catch. I had run remove_stale_pyc_files.py before testing this
> patch,
> > but forgot to add the check in.
> 
> Since this no longer block the other patch. I think we can close this issue
for
> now as we are still unsure about the best approach?

I updated the patch. Why choose one approach when you can do both? :P

[dtu, 2015-08-04 21:04:33]

Ping

[nednguyen, 2015-08-04 22:22:02]

https://codereview.chromium.org/1254173003/diff/40001/tools/telemetry/telemet...
File tools/telemetry/telemetry/internal/util/find_dependencies.py (right):

https://codereview.chromium.org/1254173003/diff/40001/tools/telemetry/telemet...
tools/telemetry/telemetry/internal/util/find_dependencies.py:41: def
FindDynamicAnalysisDependencies(module_paths):
the method name is ambiguous.

https://codereview.chromium.org/1254173003/diff/40001/tools/telemetry/telemet...
tools/telemetry/telemetry/internal/util/find_dependencies.py:45:
_LoadModules(module_paths)
Wouldn't this includes module imported by this file?

https://codereview.chromium.org/1254173003/diff/40001/tools/telemetry/telemet...
tools/telemetry/telemetry/internal/util/find_dependencies.py:48: for module in
sys.modules.itervalues():
So I guess the difference between this and other method is that this also
discovers modules that only become importable due to loading others in
module_paths?

Can you add unittest to show the differences?

https://codereview.chromium.org/1254173003/diff/40001/tools/telemetry/telemet...
tools/telemetry/telemetry/internal/util/find_dependencies.py:74: def
FindStaticAnalysisDependencies(module_path):
ditto

https://codereview.chromium.org/1254173003/diff/40001/tools/telemetry/telemet...
tools/telemetry/telemetry/internal/util/find_dependencies.py:78:
_LoadModules([module_path])
Doesn't modulefinder also execute the sys.path modification?

[nednguyen, 2016-10-10 14:27:55]

nednguyen@google.com changed reviewers:
- nednguyen@google.com