OLD | NEW |
1 #!/usr/bin/env python | 1 # DEPS Files |
2 # Copyright 2013 The Chromium Authors. All rights reserved. | |
3 # Use of this source code is governed by a BSD-style license that can be | |
4 # found in the LICENSE file. | |
5 | 2 |
6 """Traverses the source tree, parses all found DEPS files, and constructs | 3 DEPS files specify which files the sources in a directory tree may include. |
7 a dependency rule table to be used by subclasses. | |
8 | 4 |
9 The format of the deps file: | 5 ## File format |
10 | 6 |
11 First you have the normal module-level deps. These are the ones used by | 7 First you have the normal module-level deps. These are the ones used by |
12 gclient. An example would be: | 8 gclient. An example would be: |
13 | 9 |
14 deps = { | 10 ``` |
15 "base":"http://foo.bar/trunk/base" | 11 deps = { |
16 } | 12 "base":"http://foo.bar/trunk/base" |
| 13 } |
| 14 ``` |
17 | 15 |
18 DEPS files not in the top-level of a module won't need this. Then you | 16 DEPS files not in the top-level of a module won't need this. Then you have any |
19 have any additional include rules. You can add (using "+") or subtract | 17 additional include rules. You can add (using `+`) or subtract (using `-`) from |
20 (using "-") from the previously specified rules (including | 18 the previously specified rules (including module-level deps). You can also |
21 module-level deps). You can also specify a path that is allowed for | 19 specify a path that is allowed for now but that we intend to remove, using `!`; |
22 now but that we intend to remove, using "!"; this is treated the same | 20 this is treated the same as `+` when `check_deps` is run by our bots, but a |
23 as "+" when check_deps is run by our bots, but a presubmit step will | 21 presubmit step will show a warning if you add a new include of a file that is |
24 show a warning if you add a new include of a file that is only allowed | 22 only allowed by `!`. |
25 by "!". | |
26 | 23 |
27 Note that for .java files, there is currently no difference between | 24 Note that for .java files, there is currently no difference between `+` and |
28 "+" and "!", even in the presubmit step. | 25 `!`, even in the presubmit step. |
29 | 26 |
30 include_rules = [ | 27 ``` |
31 # Code should be able to use base (it's specified in the module-level | 28 include_rules = [ |
32 # deps above), but nothing in "base/evil" because it's evil. | 29 # Code should be able to use base (it's specified in the module-level |
33 "-base/evil", | 30 # deps above), but nothing in "base/evil" because it's evil. |
| 31 "-base/evil", |
34 | 32 |
35 # But this one subdirectory of evil is OK. | 33 # But this one subdirectory of evil is OK. |
36 "+base/evil/not", | 34 "+base/evil/not", |
37 | 35 |
38 # And it can include files from this other directory even though there is | 36 # And it can include files from this other directory even though there is |
39 # no deps rule for it. | 37 # no deps rule for it. |
40 "+tools/crime_fighter", | 38 "+tools/crime_fighter", |
41 | 39 |
42 # This dependency is allowed for now but work is ongoing to remove it, | 40 # This dependency is allowed for now but work is ongoing to remove it, |
43 # so you shouldn't add further dependencies on it. | 41 # so you shouldn't add further dependencies on it. |
44 "!base/evil/ok_for_now.h", | 42 "!base/evil/ok_for_now.h", |
45 ] | 43 ] |
| 44 ``` |
46 | 45 |
47 If you have certain include rules that should only be applied for some | 46 If you have certain include rules that should only be applied for some files |
48 files within this directory and subdirectories, you can write a | 47 within this directory and subdirectories, you can write a section named |
49 section named specific_include_rules that is a hash map of regular | 48 `specific_include_rules` that is a hash map of regular expressions to the list |
50 expressions to the list of rules that should apply to files matching | 49 of rules that should apply to files matching them. Note that such rules will |
51 them. Note that such rules will always be applied before the rules | 50 always be applied before the rules from `include_rules` have been applied, but |
52 from 'include_rules' have been applied, but the order in which rules | 51 the order in which rules associated with different regular expressions is |
53 associated with different regular expressions is applied is arbitrary. | 52 applied is arbitrary. |
54 | 53 |
55 specific_include_rules = { | 54 ``` |
56 ".*_(unit|browser|api)test\.cc": [ | 55 specific_include_rules = { |
57 "+libraries/testsupport", | 56 ".*_(unit|browser|api)test\.cc": [ |
58 ], | 57 "+libraries/testsupport", |
59 } | 58 ], |
| 59 } |
| 60 ``` |
| 61 |
| 62 # Directory structure |
60 | 63 |
61 DEPS files may be placed anywhere in the tree. Each one applies to all | 64 DEPS files may be placed anywhere in the tree. Each one applies to all |
62 subdirectories, where there may be more DEPS files that provide additions or | 65 subdirectories, where there may be more DEPS files that provide additions or |
63 subtractions for their own sub-trees. | 66 subtractions for their own sub-trees. |
64 | 67 |
65 There is an implicit rule for the current directory (where the DEPS file lives) | 68 There is an implicit rule for the current directory (where the DEPS file lives) |
66 and all of its subdirectories. This prevents you from having to explicitly | 69 and all of its subdirectories. This prevents you from having to explicitly |
67 allow the current directory everywhere. This implicit rule is applied first, | 70 allow the current directory everywhere. This implicit rule is applied first, so |
68 so you can modify or remove it using the normal include rules. | 71 you can modify or remove it using the normal include rules. |
69 | 72 |
70 The rules are processed in order. This means you can explicitly allow a higher | 73 The rules are processed in order. This means you can explicitly allow a higher |
71 directory and then take away permissions from sub-parts, or the reverse. | 74 directory and then take away permissions from sub-parts, or the reverse. |
72 | 75 |
73 Note that all directory separators must be slashes (Unix-style) and not | 76 Note that all directory separators must be `/` slashes (Unix-style) and not |
74 backslashes. All directories should be relative to the source root and use | 77 backslashes. All directories should be relative to the source root and use |
75 only lowercase. | 78 only lowercase. |
76 """ | |
77 | |
78 import copy | |
79 import os.path | |
80 import posixpath | |
81 import subprocess | |
82 | |
83 from rules import Rule, Rules | |
84 | |
85 | |
86 # Variable name used in the DEPS file to add or subtract include files from | |
87 # the module-level deps. | |
88 INCLUDE_RULES_VAR_NAME = 'include_rules' | |
89 | |
90 # Variable name used in the DEPS file to add or subtract include files | |
91 # from module-level deps specific to files whose basename (last | |
92 # component of path) matches a given regular expression. | |
93 SPECIFIC_INCLUDE_RULES_VAR_NAME = 'specific_include_rules' | |
94 | |
95 # Optionally present in the DEPS file to list subdirectories which should not | |
96 # be checked. This allows us to skip third party code, for example. | |
97 SKIP_SUBDIRS_VAR_NAME = 'skip_child_includes' | |
98 | |
99 | |
100 class DepsBuilderError(Exception): | |
101 """Base class for exceptions in this module.""" | |
102 pass | |
103 | |
104 | |
105 def NormalizePath(path): | |
106 """Returns a path normalized to how we write DEPS rules and compare paths.""" | |
107 return os.path.normcase(path).replace(os.path.sep, posixpath.sep) | |
108 | |
109 | |
110 def _GitSourceDirectories(base_directory): | |
111 """Returns set of normalized paths to subdirectories containing sources | |
112 managed by git.""" | |
113 base_dir_norm = NormalizePath(base_directory) | |
114 git_source_directories = set([base_dir_norm]) | |
115 | |
116 git_cmd = 'git.bat' if os.name == 'nt' else 'git' | |
117 git_ls_files_cmd = [git_cmd, 'ls-files'] | |
118 # FIXME: Use a context manager in Python 3.2+ | |
119 popen = subprocess.Popen(git_ls_files_cmd, | |
120 stdout=subprocess.PIPE, | |
121 bufsize=1, # line buffering, since read by line | |
122 cwd=base_directory) | |
123 try: | |
124 try: | |
125 for line in popen.stdout: | |
126 dir_path = os.path.join(base_directory, os.path.dirname(line)) | |
127 dir_path_norm = NormalizePath(dir_path) | |
128 # Add the directory as well as all the parent directories, | |
129 # stopping once we reach an already-listed directory. | |
130 while dir_path_norm not in git_source_directories: | |
131 git_source_directories.add(dir_path_norm) | |
132 dir_path_norm = posixpath.dirname(dir_path_norm) | |
133 finally: | |
134 popen.stdout.close() | |
135 finally: | |
136 popen.wait() | |
137 | |
138 return git_source_directories | |
139 | |
140 | |
141 class DepsBuilder(object): | |
142 """Parses include_rules from DEPS files.""" | |
143 | |
144 def __init__(self, | |
145 base_directory=None, | |
146 extra_repos=[], | |
147 verbose=False, | |
148 being_tested=False, | |
149 ignore_temp_rules=False, | |
150 ignore_specific_rules=False): | |
151 """Creates a new DepsBuilder. | |
152 | |
153 Args: | |
154 base_directory: local path to root of checkout, e.g. C:\chr\src. | |
155 verbose: Set to True for debug output. | |
156 being_tested: Set to True to ignore the DEPS file at tools/checkdeps/DEPS. | |
157 ignore_temp_rules: Ignore rules that start with Rule.TEMP_ALLOW ("!"). | |
158 """ | |
159 base_directory = (base_directory or | |
160 os.path.join(os.path.dirname(__file__), | |
161 os.path.pardir, os.path.pardir)) | |
162 self.base_directory = os.path.abspath(base_directory) # Local absolute path | |
163 self.extra_repos = extra_repos | |
164 self.verbose = verbose | |
165 self._under_test = being_tested | |
166 self._ignore_temp_rules = ignore_temp_rules | |
167 self._ignore_specific_rules = ignore_specific_rules | |
168 self._git_source_directories = None | |
169 | |
170 if os.path.exists(os.path.join(base_directory, '.git')): | |
171 self.is_git = True | |
172 elif os.path.exists(os.path.join(base_directory, '.svn')): | |
173 self.is_git = False | |
174 else: | |
175 raise DepsBuilderError("%s is not a repository root" % base_directory) | |
176 | |
177 # Map of normalized directory paths to rules to use for those | |
178 # directories, or None for directories that should be skipped. | |
179 # Normalized is: absolute, lowercase, / for separator. | |
180 self.directory_rules = {} | |
181 self._ApplyDirectoryRulesAndSkipSubdirs(Rules(), self.base_directory) | |
182 | |
183 def _ApplyRules(self, existing_rules, includes, specific_includes, | |
184 cur_dir_norm): | |
185 """Applies the given include rules, returning the new rules. | |
186 | |
187 Args: | |
188 existing_rules: A set of existing rules that will be combined. | |
189 include: The list of rules from the "include_rules" section of DEPS. | |
190 specific_includes: E.g. {'.*_unittest\.cc': ['+foo', '-blat']} rules | |
191 from the "specific_include_rules" section of DEPS. | |
192 cur_dir_norm: The current directory, normalized path. We will create an | |
193 implicit rule that allows inclusion from this directory. | |
194 | |
195 Returns: A new set of rules combining the existing_rules with the other | |
196 arguments. | |
197 """ | |
198 rules = copy.deepcopy(existing_rules) | |
199 | |
200 # First apply the implicit "allow" rule for the current directory. | |
201 base_dir_norm = NormalizePath(self.base_directory) | |
202 if not cur_dir_norm.startswith(base_dir_norm): | |
203 raise Exception( | |
204 'Internal error: base directory is not at the beginning for\n' | |
205 ' %s and base dir\n' | |
206 ' %s' % (cur_dir_norm, base_dir_norm)) | |
207 relative_dir = posixpath.relpath(cur_dir_norm, base_dir_norm) | |
208 | |
209 # Make the help string a little more meaningful. | |
210 source = relative_dir or 'top level' | |
211 rules.AddRule('+' + relative_dir, | |
212 relative_dir, | |
213 'Default rule for ' + source) | |
214 | |
215 def ApplyOneRule(rule_str, dependee_regexp=None): | |
216 """Deduces a sensible description for the rule being added, and | |
217 adds the rule with its description to |rules|. | |
218 | |
219 If we are ignoring temporary rules, this function does nothing | |
220 for rules beginning with the Rule.TEMP_ALLOW character. | |
221 """ | |
222 if self._ignore_temp_rules and rule_str.startswith(Rule.TEMP_ALLOW): | |
223 return | |
224 | |
225 rule_block_name = 'include_rules' | |
226 if dependee_regexp: | |
227 rule_block_name = 'specific_include_rules' | |
228 if relative_dir: | |
229 rule_description = relative_dir + "'s %s" % rule_block_name | |
230 else: | |
231 rule_description = 'the top level %s' % rule_block_name | |
232 rules.AddRule(rule_str, relative_dir, rule_description, dependee_regexp) | |
233 | |
234 # Apply the additional explicit rules. | |
235 for rule_str in includes: | |
236 ApplyOneRule(rule_str) | |
237 | |
238 # Finally, apply the specific rules. | |
239 if self._ignore_specific_rules: | |
240 return rules | |
241 | |
242 for regexp, specific_rules in specific_includes.iteritems(): | |
243 for rule_str in specific_rules: | |
244 ApplyOneRule(rule_str, regexp) | |
245 | |
246 return rules | |
247 | |
248 def _ApplyDirectoryRules(self, existing_rules, dir_path_local_abs): | |
249 """Combines rules from the existing rules and the new directory. | |
250 | |
251 Any directory can contain a DEPS file. Top-level DEPS files can contain | |
252 module dependencies which are used by gclient. We use these, along with | |
253 additional include rules and implicit rules for the given directory, to | |
254 come up with a combined set of rules to apply for the directory. | |
255 | |
256 Args: | |
257 existing_rules: The rules for the parent directory. We'll add-on to these. | |
258 dir_path_local_abs: The directory path that the DEPS file may live in (if | |
259 it exists). This will also be used to generate the | |
260 implicit rules. This is a local path. | |
261 | |
262 Returns: A 2-tuple of: | |
263 (1) the combined set of rules to apply to the sub-tree, | |
264 (2) a list of all subdirectories that should NOT be checked, as specified | |
265 in the DEPS file (if any). | |
266 Subdirectories are single words, hence no OS dependence. | |
267 """ | |
268 dir_path_norm = NormalizePath(dir_path_local_abs) | |
269 | |
270 # Check the DEPS file in this directory. | |
271 if self.verbose: | |
272 print 'Applying rules from', dir_path_local_abs | |
273 def FromImpl(*_): | |
274 pass # NOP function so "From" doesn't fail. | |
275 | |
276 def FileImpl(_): | |
277 pass # NOP function so "File" doesn't fail. | |
278 | |
279 class _VarImpl: | |
280 def __init__(self, local_scope): | |
281 self._local_scope = local_scope | |
282 | |
283 def Lookup(self, var_name): | |
284 """Implements the Var syntax.""" | |
285 try: | |
286 return self._local_scope['vars'][var_name] | |
287 except KeyError: | |
288 raise Exception('Var is not defined: %s' % var_name) | |
289 | |
290 local_scope = {} | |
291 global_scope = { | |
292 'File': FileImpl, | |
293 'From': FromImpl, | |
294 'Var': _VarImpl(local_scope).Lookup, | |
295 } | |
296 deps_file_path = os.path.join(dir_path_local_abs, 'DEPS') | |
297 | |
298 # The second conditional here is to disregard the | |
299 # tools/checkdeps/DEPS file while running tests. This DEPS file | |
300 # has a skip_child_includes for 'testdata' which is necessary for | |
301 # running production tests, since there are intentional DEPS | |
302 # violations under the testdata directory. On the other hand when | |
303 # running tests, we absolutely need to verify the contents of that | |
304 # directory to trigger those intended violations and see that they | |
305 # are handled correctly. | |
306 if os.path.isfile(deps_file_path) and not ( | |
307 self._under_test and | |
308 os.path.basename(dir_path_local_abs) == 'checkdeps'): | |
309 execfile(deps_file_path, global_scope, local_scope) | |
310 elif self.verbose: | |
311 print ' No deps file found in', dir_path_local_abs | |
312 | |
313 # Even if a DEPS file does not exist we still invoke ApplyRules | |
314 # to apply the implicit "allow" rule for the current directory | |
315 include_rules = local_scope.get(INCLUDE_RULES_VAR_NAME, []) | |
316 specific_include_rules = local_scope.get(SPECIFIC_INCLUDE_RULES_VAR_NAME, | |
317 {}) | |
318 skip_subdirs = local_scope.get(SKIP_SUBDIRS_VAR_NAME, []) | |
319 | |
320 return (self._ApplyRules(existing_rules, include_rules, | |
321 specific_include_rules, dir_path_norm), | |
322 skip_subdirs) | |
323 | |
324 def _ApplyDirectoryRulesAndSkipSubdirs(self, parent_rules, | |
325 dir_path_local_abs): | |
326 """Given |parent_rules| and a subdirectory |dir_path_local_abs| of the | |
327 directory that owns the |parent_rules|, add |dir_path_local_abs|'s rules to | |
328 |self.directory_rules|, and add None entries for any of its | |
329 subdirectories that should be skipped. | |
330 """ | |
331 directory_rules, excluded_subdirs = self._ApplyDirectoryRules( | |
332 parent_rules, dir_path_local_abs) | |
333 dir_path_norm = NormalizePath(dir_path_local_abs) | |
334 self.directory_rules[dir_path_norm] = directory_rules | |
335 for subdir in excluded_subdirs: | |
336 subdir_path_norm = posixpath.join(dir_path_norm, subdir) | |
337 self.directory_rules[subdir_path_norm] = None | |
338 | |
339 def GetAllRulesAndFiles(self, dir_name=None): | |
340 """Yields (rules, filenames) for each repository directory with DEPS rules. | |
341 | |
342 This walks the directory tree while staying in the repository. Specify | |
343 |dir_name| to walk just one directory and its children; omit |dir_name| to | |
344 walk the entire repository. | |
345 | |
346 Yields: | |
347 Two-element (rules, filenames) tuples. |rules| is a rules.Rules object | |
348 for a directory, and |filenames| is a list of the absolute local paths | |
349 of all files in that directory. | |
350 """ | |
351 if self.is_git and self._git_source_directories is None: | |
352 self._git_source_directories = _GitSourceDirectories(self.base_directory) | |
353 for repo in self.extra_repos: | |
354 repo_path = os.path.join(self.base_directory, repo) | |
355 self._git_source_directories.update(_GitSourceDirectories(repo_path)) | |
356 | |
357 # Collect a list of all files and directories to check. | |
358 files_to_check = [] | |
359 if dir_name and not os.path.isabs(dir_name): | |
360 dir_name = os.path.join(self.base_directory, dir_name) | |
361 dirs_to_check = [dir_name or self.base_directory] | |
362 while dirs_to_check: | |
363 current_dir = dirs_to_check.pop() | |
364 | |
365 # Check that this directory is part of the source repository. This | |
366 # prevents us from descending into third-party code or directories | |
367 # generated by the build system. | |
368 if self.is_git: | |
369 if NormalizePath(current_dir) not in self._git_source_directories: | |
370 continue | |
371 elif not os.path.exists(os.path.join(current_dir, '.svn')): | |
372 continue | |
373 | |
374 current_dir_rules = self.GetDirectoryRules(current_dir) | |
375 | |
376 if not current_dir_rules: | |
377 continue # Handle the 'skip_child_includes' case. | |
378 | |
379 current_dir_contents = sorted(os.listdir(current_dir)) | |
380 file_names = [] | |
381 sub_dirs = [] | |
382 for file_name in current_dir_contents: | |
383 full_name = os.path.join(current_dir, file_name) | |
384 if os.path.isdir(full_name): | |
385 sub_dirs.append(full_name) | |
386 else: | |
387 file_names.append(full_name) | |
388 dirs_to_check.extend(reversed(sub_dirs)) | |
389 | |
390 yield (current_dir_rules, file_names) | |
391 | |
392 def GetDirectoryRules(self, dir_path_local): | |
393 """Returns a Rules object to use for the given directory, or None | |
394 if the given directory should be skipped. | |
395 | |
396 Also modifies |self.directory_rules| to store the Rules. | |
397 This takes care of first building rules for parent directories (up to | |
398 |self.base_directory|) if needed, which may add rules for skipped | |
399 subdirectories. | |
400 | |
401 Args: | |
402 dir_path_local: A local path to the directory you want rules for. | |
403 Can be relative and unnormalized. It is the caller's responsibility | |
404 to ensure that this is part of the repository rooted at | |
405 |self.base_directory|. | |
406 """ | |
407 if os.path.isabs(dir_path_local): | |
408 dir_path_local_abs = dir_path_local | |
409 else: | |
410 dir_path_local_abs = os.path.join(self.base_directory, dir_path_local) | |
411 dir_path_norm = NormalizePath(dir_path_local_abs) | |
412 | |
413 if dir_path_norm in self.directory_rules: | |
414 return self.directory_rules[dir_path_norm] | |
415 | |
416 parent_dir_local_abs = os.path.dirname(dir_path_local_abs) | |
417 parent_rules = self.GetDirectoryRules(parent_dir_local_abs) | |
418 # We need to check for an entry for our dir_path again, since | |
419 # GetDirectoryRules can modify entries for subdirectories, namely setting | |
420 # to None if they should be skipped, via _ApplyDirectoryRulesAndSkipSubdirs. | |
421 # For example, if dir_path == 'A/B/C' and A/B/DEPS specifies that the C | |
422 # subdirectory be skipped, GetDirectoryRules('A/B') will fill in the entry | |
423 # for 'A/B/C' as None. | |
424 if dir_path_norm in self.directory_rules: | |
425 return self.directory_rules[dir_path_norm] | |
426 | |
427 if parent_rules: | |
428 self._ApplyDirectoryRulesAndSkipSubdirs(parent_rules, dir_path_local_abs) | |
429 else: | |
430 # If the parent directory should be skipped, then the current | |
431 # directory should also be skipped. | |
432 self.directory_rules[dir_path_norm] = None | |
433 return self.directory_rules[dir_path_norm] | |
OLD | NEW |