Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(57)

Side by Side Diff: tools/telemetry/third_party/subprocess32/subprocess32.py

Issue 1323303002: [Telemetry] Add timeout to telemetry run_tests (Closed) Base URL: https://chromium.googlesource.com/chromium/src.git@master
Patch Set: Created 5 years, 3 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
OLDNEW
(Empty)
1 # subprocess - Subprocesses with accessible I/O streams
2 #
3 # For more information about this module, see PEP 324.
4 #
5 # Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
6 #
7 # Licensed to PSF under a Contributor Agreement.
8 # See http://www.python.org/3.3/license for licensing details.
9
10 r"""subprocess - Subprocesses with accessible I/O streams
11
12 This module allows you to spawn processes, connect to their
13 input/output/error pipes, and obtain their return codes. This module
14 intends to replace several other, older modules and functions, like:
15
16 os.system
17 os.spawn*
18 os.popen*
19 popen2.*
20 commands.*
21
22 Information about how the subprocess module can be used to replace these
23 modules and functions can be found below.
24
25
26
27 Using the subprocess module
28 ===========================
29 This module defines one class called Popen:
30
31 class Popen(args, bufsize=0, executable=None,
32 stdin=None, stdout=None, stderr=None,
33 preexec_fn=None, close_fds=True, shell=False,
34 cwd=None, env=None, universal_newlines=False,
35 startupinfo=None, creationflags=0,
36 restore_signals=True, start_new_session=False, pass_fds=()):
37
38
39 Arguments are:
40
41 args should be a string, or a sequence of program arguments. The
42 program to execute is normally the first item in the args sequence or
43 string, but can be explicitly set by using the executable argument.
44
45 On POSIX, with shell=False (default): In this case, the Popen class
46 uses os.execvp() to execute the child program. args should normally
47 be a sequence. A string will be treated as a sequence with the string
48 as the only item (the program to execute).
49
50 On POSIX, with shell=True: If args is a string, it specifies the
51 command string to execute through the shell. If args is a sequence,
52 the first item specifies the command string, and any additional items
53 will be treated as additional shell arguments.
54
55 On Windows: the Popen class uses CreateProcess() to execute the child
56 program, which operates on strings. If args is a sequence, it will be
57 converted to a string using the list2cmdline method. Please note that
58 not all MS Windows applications interpret the command line the same
59 way: The list2cmdline is designed for applications using the same
60 rules as the MS C runtime.
61
62 bufsize, if given, has the same meaning as the corresponding argument
63 to the built-in open() function: 0 means unbuffered, 1 means line
64 buffered, any other positive value means use a buffer of
65 (approximately) that size. A negative bufsize means to use the system
66 default, which usually means fully buffered. The default value for
67 bufsize is 0 (unbuffered).
68
69 stdin, stdout and stderr specify the executed programs' standard
70 input, standard output and standard error file handles, respectively.
71 Valid values are PIPE, an existing file descriptor (a positive
72 integer), an existing file object, and None. PIPE indicates that a
73 new pipe to the child should be created. With None, no redirection
74 will occur; the child's file handles will be inherited from the
75 parent. Additionally, stderr can be STDOUT, which indicates that the
76 stderr data from the applications should be captured into the same
77 file handle as for stdout.
78
79 On POSIX, if preexec_fn is set to a callable object, this object will be
80 called in the child process just before the child is executed. The use
81 of preexec_fn is not thread safe, using it in the presence of threads
82 could lead to a deadlock in the child process before the new executable
83 is executed.
84
85 If close_fds is true, all file descriptors except 0, 1 and 2 will be
86 closed before the child process is executed. The default for close_fds
87 varies by platform: Always true on POSIX. True when stdin/stdout/stderr
88 are None on Windows, false otherwise.
89
90 pass_fds is an optional sequence of file descriptors to keep open between the
91 parent and child. Providing any pass_fds implicitly sets close_fds to true.
92
93 if shell is true, the specified command will be executed through the
94 shell.
95
96 If cwd is not None, the current directory will be changed to cwd
97 before the child is executed.
98
99 On POSIX, if restore_signals is True all signals that Python sets to
100 SIG_IGN are restored to SIG_DFL in the child process before the exec.
101 Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals. This
102 parameter does nothing on Windows.
103
104 On POSIX, if start_new_session is True, the setsid() system call will be made
105 in the child process prior to executing the command.
106
107 If env is not None, it defines the environment variables for the new
108 process.
109
110 If universal_newlines is true, the file objects stdout and stderr are
111 opened as a text files, but lines may be terminated by any of '\n',
112 the Unix end-of-line convention, '\r', the old Macintosh convention or
113 '\r\n', the Windows convention. All of these external representations
114 are seen as '\n' by the Python program. Note: This feature is only
115 available if Python is built with universal newline support (the
116 default). Also, the newlines attribute of the file objects stdout,
117 stdin and stderr are not updated by the communicate() method.
118
119 The startupinfo and creationflags, if given, will be passed to the
120 underlying CreateProcess() function. They can specify things such as
121 appearance of the main window and priority for the new process.
122 (Windows only)
123
124
125 This module also defines some shortcut functions:
126
127 call(*popenargs, **kwargs):
128 Run command with arguments. Wait for command to complete, then
129 return the returncode attribute.
130
131 The arguments are the same as for the Popen constructor. Example:
132
133 retcode = call(["ls", "-l"])
134
135 check_call(*popenargs, **kwargs):
136 Run command with arguments. Wait for command to complete. If the
137 exit code was zero then return, otherwise raise
138 CalledProcessError. The CalledProcessError object will have the
139 return code in the returncode attribute.
140
141 The arguments are the same as for the Popen constructor. Example:
142
143 check_call(["ls", "-l"])
144
145 check_output(*popenargs, **kwargs):
146 Run command with arguments and return its output as a byte string.
147
148 If the exit code was non-zero it raises a CalledProcessError. The
149 CalledProcessError object will have the return code in the returncode
150 attribute and output in the output attribute.
151
152 The arguments are the same as for the Popen constructor. Example:
153
154 output = check_output(["ls", "-l", "/dev/null"])
155
156
157 Exceptions
158 ----------
159 Exceptions raised in the child process, before the new program has
160 started to execute, will be re-raised in the parent. Additionally,
161 the exception object will have one extra attribute called
162 'child_traceback', which is a string containing traceback information
163 from the childs point of view.
164
165 The most common exception raised is OSError. This occurs, for
166 example, when trying to execute a non-existent file. Applications
167 should prepare for OSErrors.
168
169 A ValueError will be raised if Popen is called with invalid arguments.
170
171 check_call() and check_output() will raise CalledProcessError, if the
172 called process returns a non-zero return code.
173
174
175 Security
176 --------
177 Unlike some other popen functions, this implementation will never call
178 /bin/sh implicitly. This means that all characters, including shell
179 metacharacters, can safely be passed to child processes.
180
181
182 Popen objects
183 =============
184 Instances of the Popen class have the following methods:
185
186 poll()
187 Check if child process has terminated. Returns returncode
188 attribute.
189
190 wait()
191 Wait for child process to terminate. Returns returncode attribute.
192
193 communicate(input=None)
194 Interact with process: Send data to stdin. Read data from stdout
195 and stderr, until end-of-file is reached. Wait for process to
196 terminate. The optional input argument should be a string to be
197 sent to the child process, or None, if no data should be sent to
198 the child.
199
200 communicate() returns a tuple (stdout, stderr).
201
202 Note: The data read is buffered in memory, so do not use this
203 method if the data size is large or unlimited.
204
205 The following attributes are also available:
206
207 stdin
208 If the stdin argument is PIPE, this attribute is a file object
209 that provides input to the child process. Otherwise, it is None.
210
211 stdout
212 If the stdout argument is PIPE, this attribute is a file object
213 that provides output from the child process. Otherwise, it is
214 None.
215
216 stderr
217 If the stderr argument is PIPE, this attribute is file object that
218 provides error output from the child process. Otherwise, it is
219 None.
220
221 pid
222 The process ID of the child process.
223
224 returncode
225 The child return code. A None value indicates that the process
226 hasn't terminated yet. A negative value -N indicates that the
227 child was terminated by signal N (POSIX only).
228
229
230 Replacing older functions with the subprocess module
231 ====================================================
232 In this section, "a ==> b" means that b can be used as a replacement
233 for a.
234
235 Note: All functions in this section fail (more or less) silently if
236 the executed program cannot be found; this module raises an OSError
237 exception.
238
239 In the following examples, we assume that the subprocess module is
240 imported with "from subprocess import *".
241
242
243 Replacing /bin/sh shell backquote
244 ---------------------------------
245 output=`mycmd myarg`
246 ==>
247 output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
248
249
250 Replacing shell pipe line
251 -------------------------
252 output=`dmesg | grep hda`
253 ==>
254 p1 = Popen(["dmesg"], stdout=PIPE)
255 p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
256 output = p2.communicate()[0]
257
258
259 Replacing os.system()
260 ---------------------
261 sts = os.system("mycmd" + " myarg")
262 ==>
263 p = Popen("mycmd" + " myarg", shell=True)
264 pid, sts = os.waitpid(p.pid, 0)
265
266 Note:
267
268 * Calling the program through the shell is usually not required.
269
270 * It's easier to look at the returncode attribute than the
271 exitstatus.
272
273 A more real-world example would look like this:
274
275 try:
276 retcode = call("mycmd" + " myarg", shell=True)
277 if retcode < 0:
278 print >>sys.stderr, "Child was terminated by signal", -retcode
279 else:
280 print >>sys.stderr, "Child returned", retcode
281 except OSError, e:
282 print >>sys.stderr, "Execution failed:", e
283
284
285 Replacing os.spawn*
286 -------------------
287 P_NOWAIT example:
288
289 pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
290 ==>
291 pid = Popen(["/bin/mycmd", "myarg"]).pid
292
293
294 P_WAIT example:
295
296 retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
297 ==>
298 retcode = call(["/bin/mycmd", "myarg"])
299
300
301 Vector example:
302
303 os.spawnvp(os.P_NOWAIT, path, args)
304 ==>
305 Popen([path] + args[1:])
306
307
308 Environment example:
309
310 os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
311 ==>
312 Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
313
314
315 Replacing os.popen*
316 -------------------
317 pipe = os.popen("cmd", mode='r', bufsize)
318 ==>
319 pipe = Popen("cmd", shell=True, bufsize=bufsize, stdout=PIPE).stdout
320
321 pipe = os.popen("cmd", mode='w', bufsize)
322 ==>
323 pipe = Popen("cmd", shell=True, bufsize=bufsize, stdin=PIPE).stdin
324
325
326 (child_stdin, child_stdout) = os.popen2("cmd", mode, bufsize)
327 ==>
328 p = Popen("cmd", shell=True, bufsize=bufsize,
329 stdin=PIPE, stdout=PIPE, close_fds=True)
330 (child_stdin, child_stdout) = (p.stdin, p.stdout)
331
332
333 (child_stdin,
334 child_stdout,
335 child_stderr) = os.popen3("cmd", mode, bufsize)
336 ==>
337 p = Popen("cmd", shell=True, bufsize=bufsize,
338 stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
339 (child_stdin,
340 child_stdout,
341 child_stderr) = (p.stdin, p.stdout, p.stderr)
342
343
344 (child_stdin, child_stdout_and_stderr) = os.popen4("cmd", mode,
345 bufsize)
346 ==>
347 p = Popen("cmd", shell=True, bufsize=bufsize,
348 stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
349 (child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)
350
351 On Unix, os.popen2, os.popen3 and os.popen4 also accept a sequence as
352 the command to execute, in which case arguments will be passed
353 directly to the program without shell intervention. This usage can be
354 replaced as follows:
355
356 (child_stdin, child_stdout) = os.popen2(["/bin/ls", "-l"], mode,
357 bufsize)
358 ==>
359 p = Popen(["/bin/ls", "-l"], bufsize=bufsize, stdin=PIPE, stdout=PIPE)
360 (child_stdin, child_stdout) = (p.stdin, p.stdout)
361
362 Return code handling translates as follows:
363
364 pipe = os.popen("cmd", 'w')
365 ...
366 rc = pipe.close()
367 if rc is not None and rc % 256:
368 print "There were some errors"
369 ==>
370 process = Popen("cmd", 'w', shell=True, stdin=PIPE)
371 ...
372 process.stdin.close()
373 if process.wait() != 0:
374 print "There were some errors"
375
376
377 Replacing popen2.*
378 ------------------
379 (child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
380 ==>
381 p = Popen(["somestring"], shell=True, bufsize=bufsize
382 stdin=PIPE, stdout=PIPE, close_fds=True)
383 (child_stdout, child_stdin) = (p.stdout, p.stdin)
384
385 On Unix, popen2 also accepts a sequence as the command to execute, in
386 which case arguments will be passed directly to the program without
387 shell intervention. This usage can be replaced as follows:
388
389 (child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize,
390 mode)
391 ==>
392 p = Popen(["mycmd", "myarg"], bufsize=bufsize,
393 stdin=PIPE, stdout=PIPE, close_fds=True)
394 (child_stdout, child_stdin) = (p.stdout, p.stdin)
395
396 The popen2.Popen3 and popen2.Popen4 basically works as subprocess.Popen,
397 except that:
398
399 * subprocess.Popen raises an exception if the execution fails
400 * the capturestderr argument is replaced with the stderr argument.
401 * stdin=PIPE and stdout=PIPE must be specified.
402 * popen2 closes all filedescriptors by default, but you have to specify
403 close_fds=True with subprocess.Popen.
404 """
405
406 import sys
407 mswindows = (sys.platform == "win32")
408
409 import os
410 import exceptions
411 import types
412 import time
413 import traceback
414 import gc
415 import signal
416
417 # Exception classes used by this module.
418 class CalledProcessError(Exception):
419 """This exception is raised when a process run by check_call() or
420 check_output() returns a non-zero exit status.
421 The exit status will be stored in the returncode attribute;
422 check_output() will also store the output in the output attribute.
423 """
424 def __init__(self, returncode, cmd, output=None):
425 self.returncode = returncode
426 self.cmd = cmd
427 self.output = output
428 def __str__(self):
429 return "Command '%s' returned non-zero exit status %d" % (self.cmd, self .returncode)
430
431
432 class TimeoutExpired(Exception):
433 """This exception is raised when the timeout expires while waiting for a
434 child process.
435 """
436 def __init__(self, cmd, timeout, output=None):
437 self.cmd = cmd
438 self.timeout = timeout
439 self.output = output
440
441 def __str__(self):
442 return ("Command '%s' timed out after %s seconds" %
443 (self.cmd, self.timeout))
444
445
446 if mswindows:
447 import threading
448 import msvcrt
449 import _subprocess
450 class STARTUPINFO:
451 dwFlags = 0
452 hStdInput = None
453 hStdOutput = None
454 hStdError = None
455 wShowWindow = 0
456 class pywintypes:
457 error = IOError
458 else:
459 import select
460 _has_poll = hasattr(select, 'poll')
461 import errno
462 import fcntl
463 import pickle
464
465 try:
466 import _posixsubprocess
467 except ImportError:
468 _posixsubprocess = None
469 import warnings
470 warnings.warn("The _posixsubprocess module is not being used. "
471 "Child process reliability may suffer if your "
472 "program uses threads.", RuntimeWarning)
473 try:
474 import threading
475 except ImportError:
476 import dummy_threading as threading
477
478 # When select or poll has indicated that the file is writable,
479 # we can write up to _PIPE_BUF bytes without risk of blocking.
480 # POSIX defines PIPE_BUF as >= 512.
481 _PIPE_BUF = getattr(select, 'PIPE_BUF', 512)
482
483 _FD_CLOEXEC = getattr(fcntl, 'FD_CLOEXEC', 1)
484
485 def _set_cloexec(fd, cloexec):
486 old = fcntl.fcntl(fd, fcntl.F_GETFD)
487 if cloexec:
488 fcntl.fcntl(fd, fcntl.F_SETFD, old | _FD_CLOEXEC)
489 else:
490 fcntl.fcntl(fd, fcntl.F_SETFD, old & ~_FD_CLOEXEC)
491
492 if _posixsubprocess:
493 _create_pipe = _posixsubprocess.cloexec_pipe
494 else:
495 def _create_pipe():
496 fds = os.pipe()
497 _set_cloexec(fds[0], True)
498 _set_cloexec(fds[1], True)
499 return fds
500
501 __all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call",
502 "check_output", "CalledProcessError"]
503
504 if mswindows:
505 from _subprocess import (CREATE_NEW_CONSOLE, CREATE_NEW_PROCESS_GROUP,
506 STD_INPUT_HANDLE, STD_OUTPUT_HANDLE,
507 STD_ERROR_HANDLE, SW_HIDE,
508 STARTF_USESTDHANDLES, STARTF_USESHOWWINDOW)
509
510 __all__.extend(["CREATE_NEW_CONSOLE", "CREATE_NEW_PROCESS_GROUP",
511 "STD_INPUT_HANDLE", "STD_OUTPUT_HANDLE",
512 "STD_ERROR_HANDLE", "SW_HIDE",
513 "STARTF_USESTDHANDLES", "STARTF_USESHOWWINDOW"])
514 try:
515 MAXFD = os.sysconf("SC_OPEN_MAX")
516 except:
517 MAXFD = 256
518
519 # This lists holds Popen instances for which the underlying process had not
520 # exited at the time its __del__ method got called: those processes are wait()ed
521 # for synchronously from _cleanup() when a new Popen object is created, to avoid
522 # zombie processes.
523 _active = []
524
525 def _cleanup():
526 for inst in _active[:]:
527 res = inst._internal_poll(_deadstate=sys.maxint)
528 if res is not None:
529 try:
530 _active.remove(inst)
531 except ValueError:
532 # This can happen if two threads create a new Popen instance.
533 # It's harmless that it was already removed, so ignore.
534 pass
535
536 PIPE = -1
537 STDOUT = -2
538
539
540 def _eintr_retry_call(func, *args):
541 while True:
542 try:
543 return func(*args)
544 except (OSError, IOError), e:
545 if e.errno == errno.EINTR:
546 continue
547 raise
548
549
550 def _get_exec_path(env=None):
551 """Returns the sequence of directories that will be searched for the
552 named executable (similar to a shell) when launching a process.
553
554 *env* must be an environment variable dict or None. If *env* is None,
555 os.environ will be used.
556 """
557 if env is None:
558 env = os.environ
559 return env.get('PATH', os.defpath).split(os.pathsep)
560
561
562 if hasattr(os, 'get_exec_path'):
563 _get_exec_path = os.get_exec_path
564
565
566 def call(*popenargs, **kwargs):
567 """Run command with arguments. Wait for command to complete or
568 timeout, then return the returncode attribute.
569
570 The arguments are the same as for the Popen constructor. Example:
571
572 retcode = call(["ls", "-l"])
573 """
574 timeout = kwargs.pop('timeout', None)
575 p = Popen(*popenargs, **kwargs)
576 try:
577 return p.wait(timeout=timeout)
578 except TimeoutExpired:
579 p.kill()
580 p.wait()
581 raise
582
583
584 def check_call(*popenargs, **kwargs):
585 """Run command with arguments. Wait for command to complete. If
586 the exit code was zero then return, otherwise raise
587 CalledProcessError. The CalledProcessError object will have the
588 return code in the returncode attribute.
589
590 The arguments are the same as for the call function. Example:
591
592 check_call(["ls", "-l"])
593 """
594 retcode = call(*popenargs, **kwargs)
595 if retcode:
596 cmd = kwargs.get("args")
597 if cmd is None:
598 cmd = popenargs[0]
599 raise CalledProcessError(retcode, cmd)
600 return 0
601
602
603 def check_output(*popenargs, **kwargs):
604 r"""Run command with arguments and return its output as a byte string.
605
606 If the exit code was non-zero it raises a CalledProcessError. The
607 CalledProcessError object will have the return code in the returncode
608 attribute and output in the output attribute.
609
610 The arguments are the same as for the Popen constructor. Example:
611
612 >>> check_output(["ls", "-l", "/dev/null"])
613 'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n'
614
615 The stdout argument is not allowed as it is used internally.
616 To capture standard error in the result, use stderr=STDOUT.
617
618 >>> check_output(["/bin/sh", "-c",
619 ... "ls -l non_existent_file ; exit 0"],
620 ... stderr=STDOUT)
621 'ls: non_existent_file: No such file or directory\n'
622 """
623 timeout = kwargs.pop('timeout', None)
624 if 'stdout' in kwargs:
625 raise ValueError('stdout argument not allowed, it will be overridden.')
626 process = Popen(stdout=PIPE, *popenargs, **kwargs)
627 try:
628 output, unused_err = process.communicate(timeout=timeout)
629 except TimeoutExpired:
630 process.kill()
631 output, unused_err = process.communicate()
632 raise TimeoutExpired(process.args, timeout, output=output)
633 retcode = process.poll()
634 if retcode:
635 raise CalledProcessError(retcode, process.args, output=output)
636 return output
637
638
639 def list2cmdline(seq):
640 """
641 Translate a sequence of arguments into a command line
642 string, using the same rules as the MS C runtime:
643
644 1) Arguments are delimited by white space, which is either a
645 space or a tab.
646
647 2) A string surrounded by double quotation marks is
648 interpreted as a single argument, regardless of white space
649 contained within. A quoted string can be embedded in an
650 argument.
651
652 3) A double quotation mark preceded by a backslash is
653 interpreted as a literal double quotation mark.
654
655 4) Backslashes are interpreted literally, unless they
656 immediately precede a double quotation mark.
657
658 5) If backslashes immediately precede a double quotation mark,
659 every pair of backslashes is interpreted as a literal
660 backslash. If the number of backslashes is odd, the last
661 backslash escapes the next double quotation mark as
662 described in rule 3.
663 """
664
665 # See
666 # http://msdn.microsoft.com/en-us/library/17w5ykft.aspx
667 # or search http://msdn.microsoft.com for
668 # "Parsing C++ Command-Line Arguments"
669 result = []
670 needquote = False
671 for arg in seq:
672 bs_buf = []
673
674 # Add a space to separate this argument from the others
675 if result:
676 result.append(' ')
677
678 needquote = (" " in arg) or ("\t" in arg) or not arg
679 if needquote:
680 result.append('"')
681
682 for c in arg:
683 if c == '\\':
684 # Don't know if we need to double yet.
685 bs_buf.append(c)
686 elif c == '"':
687 # Double backslashes.
688 result.append('\\' * len(bs_buf)*2)
689 bs_buf = []
690 result.append('\\"')
691 else:
692 # Normal char
693 if bs_buf:
694 result.extend(bs_buf)
695 bs_buf = []
696 result.append(c)
697
698 # Add remaining backslashes, if any.
699 if bs_buf:
700 result.extend(bs_buf)
701
702 if needquote:
703 result.extend(bs_buf)
704 result.append('"')
705
706 return ''.join(result)
707
708
709 _PLATFORM_DEFAULT_CLOSE_FDS = object()
710
711
712 class Popen(object):
713 def __init__(self, args, bufsize=0, executable=None,
714 stdin=None, stdout=None, stderr=None,
715 preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS,
716 shell=False, cwd=None, env=None, universal_newlines=False,
717 startupinfo=None, creationflags=0,
718 restore_signals=True, start_new_session=False,
719 pass_fds=()):
720 """Create new Popen instance."""
721 _cleanup()
722 # Held while anything is calling waitpid before returncode has been
723 # updated to prevent clobbering returncode if wait() or poll() are
724 # called from multiple threads at once. After acquiring the lock,
725 # code must re-check self.returncode to see if another thread just
726 # finished a waitpid() call.
727 self._waitpid_lock = threading.Lock()
728
729 self._child_created = False
730 self._input = None
731 self._communication_started = False
732 if not isinstance(bufsize, (int, long)):
733 raise TypeError("bufsize must be an integer")
734
735 if mswindows:
736 if preexec_fn is not None:
737 raise ValueError("preexec_fn is not supported on Windows "
738 "platforms")
739 any_stdio_set = (stdin is not None or stdout is not None or
740 stderr is not None)
741 if close_fds is _PLATFORM_DEFAULT_CLOSE_FDS:
742 if any_stdio_set:
743 close_fds = False
744 else:
745 close_fds = True
746 elif close_fds and any_stdio_set:
747 raise ValueError(
748 "close_fds is not supported on Windows platforms"
749 " if you redirect stdin/stdout/stderr")
750 else:
751 # POSIX
752 if close_fds is _PLATFORM_DEFAULT_CLOSE_FDS:
753 close_fds = True
754 if pass_fds and not close_fds:
755 warnings.warn("pass_fds overriding close_fds.", RuntimeWarning)
756 close_fds = True
757 if startupinfo is not None:
758 raise ValueError("startupinfo is only supported on Windows "
759 "platforms")
760 if creationflags != 0:
761 raise ValueError("creationflags is only supported on Windows "
762 "platforms")
763
764 self.args = args
765 self.stdin = None
766 self.stdout = None
767 self.stderr = None
768 self.pid = None
769 self.returncode = None
770 self.universal_newlines = universal_newlines
771
772 # Input and output objects. The general principle is like
773 # this:
774 #
775 # Parent Child
776 # ------ -----
777 # p2cwrite ---stdin---> p2cread
778 # c2pread <--stdout--- c2pwrite
779 # errread <--stderr--- errwrite
780 #
781 # On POSIX, the child objects are file descriptors. On
782 # Windows, these are Windows file handles. The parent objects
783 # are file descriptors on both platforms. The parent objects
784 # are -1 when not using PIPEs. The child objects are -1
785 # when not redirecting.
786
787 (p2cread, p2cwrite,
788 c2pread, c2pwrite,
789 errread, errwrite) = self._get_handles(stdin, stdout, stderr)
790
791 if mswindows:
792 if p2cwrite != -1:
793 p2cwrite = msvcrt.open_osfhandle(p2cwrite.Detach(), 0)
794 if c2pread != -1:
795 c2pread = msvcrt.open_osfhandle(c2pread.Detach(), 0)
796 if errread != -1:
797 errread = msvcrt.open_osfhandle(errread.Detach(), 0)
798
799 if p2cwrite != -1:
800 self.stdin = os.fdopen(p2cwrite, 'wb', bufsize)
801 if c2pread != -1:
802 if universal_newlines:
803 self.stdout = os.fdopen(c2pread, 'rU', bufsize)
804 else:
805 self.stdout = os.fdopen(c2pread, 'rb', bufsize)
806 if errread != -1:
807 if universal_newlines:
808 self.stderr = os.fdopen(errread, 'rU', bufsize)
809 else:
810 self.stderr = os.fdopen(errread, 'rb', bufsize)
811
812 self._closed_child_pipe_fds = False
813 exception_cleanup_needed = False
814 try:
815 try:
816 self._execute_child(args, executable, preexec_fn, close_fds,
817 pass_fds, cwd, env, universal_newlines,
818 startupinfo, creationflags, shell,
819 p2cread, p2cwrite,
820 c2pread, c2pwrite,
821 errread, errwrite,
822 restore_signals, start_new_session)
823 except:
824 # The cleanup is performed within the finally block rather
825 # than simply within this except block before the raise so
826 # that any exceptions raised and handled within it do not
827 # clobber the exception context we want to propagate upwards.
828 # This is only necessary in Python 2.
829 exception_cleanup_needed = True
830 raise
831 finally:
832 if exception_cleanup_needed:
833 for f in filter(None, (self.stdin, self.stdout, self.stderr)):
834 try:
835 f.close()
836 except EnvironmentError:
837 pass # Ignore EBADF or other errors
838
839 if not self._closed_child_pipe_fds:
840 to_close = []
841 if stdin == PIPE:
842 to_close.append(p2cread)
843 if stdout == PIPE:
844 to_close.append(c2pwrite)
845 if stderr == PIPE:
846 to_close.append(errwrite)
847 for fd in to_close:
848 try:
849 os.close(fd)
850 except EnvironmentError:
851 pass
852
853 def __enter__(self):
854 return self
855
856 def __exit__(self, type, value, traceback):
857 if self.stdout:
858 self.stdout.close()
859 if self.stderr:
860 self.stderr.close()
861 if self.stdin:
862 self.stdin.close()
863 # Wait for the process to terminate, to avoid zombies.
864 self.wait()
865
866 def _translate_newlines(self, data):
867 data = data.replace("\r\n", "\n")
868 data = data.replace("\r", "\n")
869 return data
870
871
872 def __del__(self, _maxint=sys.maxint, _active=_active):
873 # If __init__ hasn't had a chance to execute (e.g. if it
874 # was passed an undeclared keyword argument), we don't
875 # have a _child_created attribute at all.
876 if not getattr(self, '_child_created', False):
877 # We didn't get to successfully create a child process.
878 return
879 # In case the child hasn't been waited on, check if it's done.
880 self._internal_poll(_deadstate=_maxint)
881 if self.returncode is None and _active is not None:
882 # Child is still running, keep us alive until we can wait on it.
883 _active.append(self)
884
885
886 def communicate(self, input=None, timeout=None):
887 """Interact with process: Send data to stdin. Read data from
888 stdout and stderr, until end-of-file is reached. Wait for
889 process to terminate. The optional input argument should be a
890 string to be sent to the child process, or None, if no data
891 should be sent to the child.
892
893 communicate() returns a tuple (stdout, stderr)."""
894
895 if self._communication_started and input:
896 raise ValueError("Cannot send input after starting communication")
897
898 if timeout is not None:
899 endtime = time.time() + timeout
900 else:
901 endtime = None
902
903 # Optimization: If we are not worried about timeouts, we haven't
904 # started communicating, and we have one or zero pipes, using select()
905 # or threads is unnecessary.
906 if (endtime is None and not self._communication_started and
907 [self.stdin, self.stdout, self.stderr].count(None) >= 2):
908 stdout = None
909 stderr = None
910 if self.stdin:
911 if input:
912 self.stdin.write(input)
913 self.stdin.close()
914 elif self.stdout:
915 stdout = _eintr_retry_call(self.stdout.read)
916 self.stdout.close()
917 elif self.stderr:
918 stderr = _eintr_retry_call(self.stderr.read)
919 self.stderr.close()
920 self.wait()
921 return (stdout, stderr)
922
923 try:
924 stdout, stderr = self._communicate(input, endtime, timeout)
925 finally:
926 self._communication_started = True
927
928 sts = self.wait(timeout=self._remaining_time(endtime))
929
930 return (stdout, stderr)
931
932
933 def poll(self):
934 return self._internal_poll()
935
936
937 def _remaining_time(self, endtime):
938 """Convenience for _communicate when computing timeouts."""
939 if endtime is None:
940 return None
941 else:
942 return endtime - time.time()
943
944
945 def _check_timeout(self, endtime, orig_timeout):
946 """Convenience for checking if a timeout has expired."""
947 if endtime is None:
948 return
949 if time.time() > endtime:
950 raise TimeoutExpired(self.args, orig_timeout)
951
952
953 if mswindows:
954 #
955 # Windows methods
956 #
957 def _get_handles(self, stdin, stdout, stderr):
958 """Construct and return tuple with IO objects:
959 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
960 """
961 if stdin is None and stdout is None and stderr is None:
962 return (-1, -1, -1, -1, -1, -1)
963
964 p2cread, p2cwrite = -1, -1
965 c2pread, c2pwrite = -1, -1
966 errread, errwrite = -1, -1
967
968 if stdin is None:
969 p2cread = _subprocess.GetStdHandle(_subprocess.STD_INPUT_HANDLE)
970 if p2cread is None:
971 p2cread, _ = _subprocess.CreatePipe(None, 0)
972 elif stdin == PIPE:
973 p2cread, p2cwrite = _subprocess.CreatePipe(None, 0)
974 elif isinstance(stdin, int):
975 p2cread = msvcrt.get_osfhandle(stdin)
976 else:
977 # Assuming file-like object
978 p2cread = msvcrt.get_osfhandle(stdin.fileno())
979 p2cread = self._make_inheritable(p2cread)
980
981 if stdout is None:
982 c2pwrite = _subprocess.GetStdHandle(_subprocess.STD_OUTPUT_HANDL E)
983 if c2pwrite is None:
984 _, c2pwrite = _subprocess.CreatePipe(None, 0)
985 elif stdout == PIPE:
986 c2pread, c2pwrite = _subprocess.CreatePipe(None, 0)
987 elif isinstance(stdout, int):
988 c2pwrite = msvcrt.get_osfhandle(stdout)
989 else:
990 # Assuming file-like object
991 c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
992 c2pwrite = self._make_inheritable(c2pwrite)
993
994 if stderr is None:
995 errwrite = _subprocess.GetStdHandle(_subprocess.STD_ERROR_HANDLE )
996 if errwrite is None:
997 _, errwrite = _subprocess.CreatePipe(None, 0)
998 elif stderr == PIPE:
999 errread, errwrite = _subprocess.CreatePipe(None, 0)
1000 elif stderr == STDOUT:
1001 errwrite = c2pwrite
1002 elif isinstance(stderr, int):
1003 errwrite = msvcrt.get_osfhandle(stderr)
1004 else:
1005 # Assuming file-like object
1006 errwrite = msvcrt.get_osfhandle(stderr.fileno())
1007 errwrite = self._make_inheritable(errwrite)
1008
1009 return (p2cread, p2cwrite,
1010 c2pread, c2pwrite,
1011 errread, errwrite)
1012
1013
1014 def _make_inheritable(self, handle):
1015 """Return a duplicate of handle, which is inheritable"""
1016 return _subprocess.DuplicateHandle(_subprocess.GetCurrentProcess(),
1017 handle, _subprocess.GetCurrentProcess(), 0, 1,
1018 _subprocess.DUPLICATE_SAME_ACCESS)
1019
1020
1021 def _find_w9xpopen(self):
1022 """Find and return absolut path to w9xpopen.exe"""
1023 w9xpopen = os.path.join(
1024 os.path.dirname(_subprocess.GetModuleFileName(0)),
1025 "w9xpopen.exe")
1026 if not os.path.exists(w9xpopen):
1027 # Eeek - file-not-found - possibly an embedding
1028 # situation - see if we can locate it in sys.exec_prefix
1029 w9xpopen = os.path.join(os.path.dirname(sys.exec_prefix),
1030 "w9xpopen.exe")
1031 if not os.path.exists(w9xpopen):
1032 raise RuntimeError("Cannot locate w9xpopen.exe, which is "
1033 "needed for Popen to work with your "
1034 "shell or platform.")
1035 return w9xpopen
1036
1037
1038 def _execute_child(self, args, executable, preexec_fn, close_fds,
1039 pass_fds, cwd, env, universal_newlines,
1040 startupinfo, creationflags, shell,
1041 p2cread, p2cwrite,
1042 c2pread, c2pwrite,
1043 errread, errwrite,
1044 unused_restore_signals, unused_start_new_session):
1045 """Execute program (MS Windows version)"""
1046
1047 assert not pass_fds, "pass_fds not supported on Windows."
1048
1049 if not isinstance(args, types.StringTypes):
1050 args = list2cmdline(args)
1051
1052 # Process startup details
1053 if startupinfo is None:
1054 startupinfo = STARTUPINFO()
1055 if -1 not in (p2cread, c2pwrite, errwrite):
1056 startupinfo.dwFlags |= _subprocess.STARTF_USESTDHANDLES
1057 startupinfo.hStdInput = p2cread
1058 startupinfo.hStdOutput = c2pwrite
1059 startupinfo.hStdError = errwrite
1060
1061 if shell:
1062 startupinfo.dwFlags |= _subprocess.STARTF_USESHOWWINDOW
1063 startupinfo.wShowWindow = _subprocess.SW_HIDE
1064 comspec = os.environ.get("COMSPEC", "cmd.exe")
1065 args = comspec + " /c " + '"%s"' % args
1066 if (_subprocess.GetVersion() >= 0x80000000L or
1067 os.path.basename(comspec).lower() == "command.com"):
1068 # Win9x, or using command.com on NT. We need to
1069 # use the w9xpopen intermediate program. For more
1070 # information, see KB Q150956
1071 # (http://web.archive.org/web/20011105084002/http://support. microsoft.com/support/kb/articles/Q150/9/56.asp)
1072 w9xpopen = self._find_w9xpopen()
1073 args = '"%s" %s' % (w9xpopen, args)
1074 # Not passing CREATE_NEW_CONSOLE has been known to
1075 # cause random failures on win9x. Specifically a
1076 # dialog: "Your program accessed mem currently in
1077 # use at xxx" and a hopeful warning about the
1078 # stability of your system. Cost is Ctrl+C wont
1079 # kill children.
1080 creationflags |= _subprocess.CREATE_NEW_CONSOLE
1081
1082 # Start the process
1083 try:
1084 try:
1085 hp, ht, pid, tid = _subprocess.CreateProcess(executable, arg s,
1086 # no special security
1087 None, None,
1088 int(not close_fds),
1089 creationflags,
1090 env,
1091 cwd,
1092 startupinfo)
1093 except pywintypes.error, e:
1094 # Translate pywintypes.error to WindowsError, which is
1095 # a subclass of OSError. FIXME: We should really
1096 # translate errno using _sys_errlist (or similar), but
1097 # how can this be done from Python?
1098 raise WindowsError(*e.args)
1099 finally:
1100 # Child is launched. Close the parent's copy of those pipe
1101 # handles that only the child should have open. You need
1102 # to make sure that no handles to the write end of the
1103 # output pipe are maintained in this process or else the
1104 # pipe will not close when the child process exits and the
1105 # ReadFile will hang.
1106 if p2cread != -1:
1107 p2cread.Close()
1108 if c2pwrite != -1:
1109 c2pwrite.Close()
1110 if errwrite != -1:
1111 errwrite.Close()
1112
1113 # Retain the process handle, but close the thread handle
1114 self._child_created = True
1115 self._handle = hp
1116 self.pid = pid
1117 ht.Close()
1118
1119 def _internal_poll(self, _deadstate=None,
1120 _WaitForSingleObject=_subprocess.WaitForSingleObject,
1121 _WAIT_OBJECT_0=_subprocess.WAIT_OBJECT_0,
1122 _GetExitCodeProcess=_subprocess.GetExitCodeProcess):
1123 """Check if child process has terminated. Returns returncode
1124 attribute.
1125
1126 This method is called by __del__, so it can only refer to objects
1127 in its local scope.
1128
1129 """
1130 if self.returncode is None:
1131 if _WaitForSingleObject(self._handle, 0) == _WAIT_OBJECT_0:
1132 self.returncode = _GetExitCodeProcess(self._handle)
1133 return self.returncode
1134
1135
1136 def wait(self, timeout=None, endtime=None):
1137 """Wait for child process to terminate. Returns returncode
1138 attribute."""
1139 if endtime is not None:
1140 timeout = self._remaining_time(endtime)
1141 if timeout is None:
1142 timeout = _subprocess.INFINITE
1143 else:
1144 timeout = int(timeout * 1000)
1145 if self.returncode is None:
1146 result = _subprocess.WaitForSingleObject(self._handle, timeout)
1147 if result == _subprocess.WAIT_TIMEOUT:
1148 raise TimeoutExpired(self.args, timeout)
1149 self.returncode = _subprocess.GetExitCodeProcess(self._handle)
1150 return self.returncode
1151
1152
1153 def _readerthread(self, fh, buffer):
1154 buffer.append(fh.read())
1155 fh.close()
1156
1157
1158 def _communicate(self, input, endtime, orig_timeout):
1159 # Start reader threads feeding into a list hanging off of this
1160 # object, unless they've already been started.
1161 if self.stdout and not hasattr(self, "_stdout_buff"):
1162 self._stdout_buff = []
1163 self.stdout_thread = \
1164 threading.Thread(target=self._readerthread,
1165 args=(self.stdout, self._stdout_buff))
1166 self.stdout_thread.daemon = True
1167 self.stdout_thread.start()
1168 if self.stderr and not hasattr(self, "_stderr_buff"):
1169 self._stderr_buff = []
1170 self.stderr_thread = \
1171 threading.Thread(target=self._readerthread,
1172 args=(self.stderr, self._stderr_buff))
1173 self.stderr_thread.daemon = True
1174 self.stderr_thread.start()
1175
1176 if self.stdin:
1177 if input is not None:
1178 self.stdin.write(input)
1179 self.stdin.close()
1180
1181 # Wait for the reader threads, or time out. If we time out, the
1182 # threads remain reading and the fds left open in case the user
1183 # calls communicate again.
1184 if self.stdout is not None:
1185 self.stdout_thread.join(self._remaining_time(endtime))
1186 if self.stdout_thread.isAlive():
1187 raise TimeoutExpired(self.args)
1188 if self.stderr is not None:
1189 self.stderr_thread.join(self._remaining_time(endtime))
1190 if self.stderr_thread.isAlive():
1191 raise TimeoutExpired(self.args)
1192
1193 # Collect the output from and close both pipes, now that we know
1194 # both have been read successfully.
1195 stdout = None
1196 stderr = None
1197 if self.stdout:
1198 stdout = self._stdout_buff
1199 self.stdout.close()
1200 if self.stderr:
1201 stderr = self._stderr_buff
1202 self.stderr.close()
1203
1204 # All data exchanged. Translate lists into strings.
1205 if stdout is not None:
1206 stdout = stdout[0]
1207 if stderr is not None:
1208 stderr = stderr[0]
1209
1210 # Translate newlines, if requested. We cannot let the file
1211 # object do the translation: It is based on stdio, which is
1212 # impossible to combine with select (unless forcing no
1213 # buffering).
1214 if self.universal_newlines and hasattr(file, 'newlines'):
1215 if stdout:
1216 stdout = self._translate_newlines(stdout)
1217 if stderr:
1218 stderr = self._translate_newlines(stderr)
1219
1220 return (stdout, stderr)
1221
1222 def send_signal(self, sig):
1223 """Send a signal to the process
1224 """
1225 if sig == signal.SIGTERM:
1226 self.terminate()
1227 elif sig == signal.CTRL_C_EVENT:
1228 os.kill(self.pid, signal.CTRL_C_EVENT)
1229 elif sig == signal.CTRL_BREAK_EVENT:
1230 os.kill(self.pid, signal.CTRL_BREAK_EVENT)
1231 else:
1232 raise ValueError("Unsupported signal: %s" % sig)
1233
1234 def terminate(self):
1235 """Terminates the process
1236 """
1237 _subprocess.TerminateProcess(self._handle, 1)
1238
1239 kill = terminate
1240
1241 else:
1242 #
1243 # POSIX methods
1244 #
1245 def _get_handles(self, stdin, stdout, stderr):
1246 """Construct and return tuple with IO objects:
1247 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
1248 """
1249 p2cread, p2cwrite = -1, -1
1250 c2pread, c2pwrite = -1, -1
1251 errread, errwrite = -1, -1
1252
1253 if stdin is None:
1254 pass
1255 elif stdin == PIPE:
1256 p2cread, p2cwrite = _create_pipe()
1257 elif isinstance(stdin, int):
1258 p2cread = stdin
1259 else:
1260 # Assuming file-like object
1261 p2cread = stdin.fileno()
1262
1263 if stdout is None:
1264 pass
1265 elif stdout == PIPE:
1266 c2pread, c2pwrite = _create_pipe()
1267 elif isinstance(stdout, int):
1268 c2pwrite = stdout
1269 else:
1270 # Assuming file-like object
1271 c2pwrite = stdout.fileno()
1272
1273 if stderr is None:
1274 pass
1275 elif stderr == PIPE:
1276 errread, errwrite = _create_pipe()
1277 elif stderr == STDOUT:
1278 errwrite = c2pwrite
1279 elif isinstance(stderr, int):
1280 errwrite = stderr
1281 else:
1282 # Assuming file-like object
1283 errwrite = stderr.fileno()
1284
1285 return (p2cread, p2cwrite,
1286 c2pread, c2pwrite,
1287 errread, errwrite)
1288
1289
1290 if hasattr(os, 'closerange'): # Introduced in 2.6
1291 @staticmethod
1292 def _closerange(fd_low, fd_high):
1293 os.closerange(fd_low, fd_high)
1294 else:
1295 @staticmethod
1296 def _closerange(fd_low, fd_high):
1297 for fd in xrange(fd_low, fd_high):
1298 while True:
1299 try:
1300 os.close(fd)
1301 except (OSError, IOError), e:
1302 if e.errno == errno.EINTR:
1303 continue
1304 break
1305
1306
1307 def _close_fds(self, but):
1308 self._closerange(3, but)
1309 self._closerange(but + 1, MAXFD)
1310
1311
1312 def _close_all_but_a_sorted_few_fds(self, fds_to_keep):
1313 # precondition: fds_to_keep must be sorted and unique
1314 start_fd = 3
1315 for fd in fds_to_keep:
1316 if fd >= start_fd:
1317 self._closerange(start_fd, fd)
1318 start_fd = fd + 1
1319 if start_fd <= MAXFD:
1320 self._closerange(start_fd, MAXFD)
1321
1322
1323 def _execute_child(self, args, executable, preexec_fn, close_fds,
1324 pass_fds, cwd, env, universal_newlines,
1325 startupinfo, creationflags, shell,
1326 p2cread, p2cwrite,
1327 c2pread, c2pwrite,
1328 errread, errwrite,
1329 restore_signals, start_new_session):
1330 """Execute program (POSIX version)"""
1331
1332 if isinstance(args, types.StringTypes):
1333 args = [args]
1334 else:
1335 args = list(args)
1336
1337 if shell:
1338 args = ["/bin/sh", "-c"] + args
1339 if executable:
1340 args[0] = executable
1341
1342 if executable is None:
1343 executable = args[0]
1344 orig_executable = executable
1345
1346 # For transferring possible exec failure from child to parent.
1347 # Data format: "exception name:hex errno:description"
1348 # Pickle is not used; it is complex and involves memory allocation.
1349 errpipe_read, errpipe_write = _create_pipe()
1350 try:
1351 try:
1352
1353 if _posixsubprocess:
1354 fs_encoding = sys.getfilesystemencoding()
1355 def fs_encode(s):
1356 """Encode s for use in the env, fs or cmdline."""
1357 if isinstance(s, str):
1358 return s
1359 else:
1360 return s.encode(fs_encoding, 'strict')
1361
1362 # We must avoid complex work that could involve
1363 # malloc or free in the child process to avoid
1364 # potential deadlocks, thus we do all this here.
1365 # and pass it to fork_exec()
1366
1367 if env is not None:
1368 env_list = [fs_encode(k) + '=' + fs_encode(v)
1369 for k, v in env.items()]
1370 else:
1371 env_list = None # Use execv instead of execve.
1372 if os.path.dirname(executable):
1373 executable_list = (fs_encode(executable),)
1374 else:
1375 # This matches the behavior of os._execvpe().
1376 path_list = _get_exec_path(env)
1377 executable_list = (os.path.join(dir, executable)
1378 for dir in path_list)
1379 executable_list = tuple(fs_encode(exe)
1380 for exe in executable_list)
1381 fds_to_keep = set(pass_fds)
1382 fds_to_keep.add(errpipe_write)
1383 self.pid = _posixsubprocess.fork_exec(
1384 args, executable_list,
1385 close_fds, sorted(fds_to_keep), cwd, env_list,
1386 p2cread, p2cwrite, c2pread, c2pwrite,
1387 errread, errwrite,
1388 errpipe_read, errpipe_write,
1389 restore_signals, start_new_session, preexec_fn)
1390 self._child_created = True
1391 else:
1392 # Pure Python implementation: It is not thread safe.
1393 # This implementation may deadlock in the child if your
1394 # parent process has any other threads running.
1395
1396 gc_was_enabled = gc.isenabled()
1397 # Disable gc to avoid bug where gc -> file_dealloc ->
1398 # write to stderr -> hang. See issue1336
1399 gc.disable()
1400 try:
1401 self.pid = os.fork()
1402 except:
1403 if gc_was_enabled:
1404 gc.enable()
1405 raise
1406 self._child_created = True
1407 if self.pid == 0:
1408 # Child
1409 reached_preexec = False
1410 try:
1411 # Close parent's pipe ends
1412 if p2cwrite != -1:
1413 os.close(p2cwrite)
1414 if c2pread != -1:
1415 os.close(c2pread)
1416 if errread != -1:
1417 os.close(errread)
1418 os.close(errpipe_read)
1419
1420 # When duping fds, if there arises a situation
1421 # where one of the fds is either 0, 1 or 2, it
1422 # is possible that it is overwritten (#12607).
1423 if c2pwrite == 0:
1424 c2pwrite = os.dup(c2pwrite)
1425 if errwrite == 0 or errwrite == 1:
1426 errwrite = os.dup(errwrite)
1427
1428 # Dup fds for child
1429 def _dup2(a, b):
1430 # dup2() removes the CLOEXEC flag but
1431 # we must do it ourselves if dup2()
1432 # would be a no-op (issue #10806).
1433 if a == b:
1434 _set_cloexec(a, False)
1435 elif a != -1:
1436 os.dup2(a, b)
1437 _dup2(p2cread, 0)
1438 _dup2(c2pwrite, 1)
1439 _dup2(errwrite, 2)
1440
1441 # Close pipe fds. Make sure we don't close the
1442 # same fd more than once, or standard fds.
1443 closed = set()
1444 for fd in [p2cread, c2pwrite, errwrite]:
1445 if fd > 2 and fd not in closed:
1446 os.close(fd)
1447 closed.add(fd)
1448
1449 if cwd is not None:
1450 os.chdir(cwd)
1451
1452 # This is a copy of Python/pythonrun.c
1453 # _Py_RestoreSignals(). If that were exposed
1454 # as a sys._py_restoresignals func it would be
1455 # better.. but this pure python implementation
1456 # isn't likely to be used much anymore.
1457 if restore_signals:
1458 signals = ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ')
1459 for sig in signals:
1460 if hasattr(signal, sig):
1461 signal.signal(getattr(signal, sig),
1462 signal.SIG_DFL)
1463
1464 if start_new_session and hasattr(os, 'setsid'):
1465 os.setsid()
1466
1467 reached_preexec = True
1468 if preexec_fn:
1469 preexec_fn()
1470
1471 # Close all other fds, if asked for - after
1472 # preexec_fn(), which may open FDs.
1473 if close_fds:
1474 if pass_fds:
1475 fds_to_keep = set(pass_fds)
1476 fds_to_keep.add(errpipe_write)
1477 self._close_all_but_a_sorted_few_fds(
1478 sorted(fds_to_keep))
1479 else:
1480 self._close_fds(but=errpipe_write)
1481
1482 if env is None:
1483 os.execvp(executable, args)
1484 else:
1485 os.execvpe(executable, args, env)
1486
1487 except:
1488 try:
1489 exc_type, exc_value = sys.exc_info()[:2]
1490 if isinstance(exc_value, OSError):
1491 errno_num = exc_value.errno
1492 else:
1493 errno_num = 0
1494 if not reached_preexec:
1495 exc_value = "noexec"
1496 message = '%s:%x:%s' % (exc_type.__name__,
1497 errno_num, exc_value )
1498 os.write(errpipe_write, message)
1499 except Exception:
1500 # We MUST not allow anything odd happening
1501 # above to prevent us from exiting below.
1502 pass
1503
1504 # This exitcode won't be reported to applications
1505 # so it really doesn't matter what we return.
1506 os._exit(255)
1507
1508 # Parent
1509 if gc_was_enabled:
1510 gc.enable()
1511 finally:
1512 # be sure the FD is closed no matter what
1513 os.close(errpipe_write)
1514
1515 # A pair of non -1s means we created both fds and are
1516 # responsible for closing them.
1517 if p2cread != -1 and p2cwrite != -1:
1518 os.close(p2cread)
1519 if c2pwrite != -1 and c2pread != -1:
1520 os.close(c2pwrite)
1521 if errwrite != -1 and errread != -1:
1522 os.close(errwrite)
1523 # Prevent a double close of these fds from __init__ on error.
1524 self._closed_child_pipe_fds = True
1525
1526 # Wait for exec to fail or succeed; possibly raising exception
1527 # exception (limited in size)
1528 errpipe_data = ''
1529 while True:
1530 part = _eintr_retry_call(os.read, errpipe_read, 50000)
1531 errpipe_data += part
1532 if not part or len(errpipe_data) > 50000:
1533 break
1534 finally:
1535 # be sure the FD is closed no matter what
1536 os.close(errpipe_read)
1537
1538 if errpipe_data != "":
1539 try:
1540 _eintr_retry_call(os.waitpid, self.pid, 0)
1541 except OSError, e:
1542 if e.errno != errno.ECHILD:
1543 raise
1544 try:
1545 exception_name, hex_errno, err_msg = (
1546 errpipe_data.split(':', 2))
1547 except ValueError:
1548 exception_name = 'RuntimeError'
1549 hex_errno = '0'
1550 err_msg = ('Bad exception data from child: ' +
1551 repr(errpipe_data))
1552 child_exception_type = getattr(
1553 exceptions, exception_name, RuntimeError)
1554 if issubclass(child_exception_type, OSError) and hex_errno:
1555 errno_num = int(hex_errno, 16)
1556 child_exec_never_called = (err_msg == "noexec")
1557 if child_exec_never_called:
1558 err_msg = ""
1559 if errno_num != 0:
1560 err_msg = os.strerror(errno_num)
1561 if errno_num == errno.ENOENT:
1562 if child_exec_never_called:
1563 # The error must be from chdir(cwd).
1564 err_msg += ': ' + repr(cwd)
1565 else:
1566 err_msg += ': ' + repr(orig_executable)
1567 raise child_exception_type(errno_num, err_msg)
1568 try:
1569 exception = child_exception_type(err_msg)
1570 except Exception:
1571 exception = RuntimeError(
1572 'Could not re-raise %r exception from the'
1573 ' child with error message %r' %
1574 (child_exception_type, err_msg))
1575 raise exception
1576
1577
1578 def _handle_exitstatus(self, sts, _WIFSIGNALED=os.WIFSIGNALED,
1579 _WTERMSIG=os.WTERMSIG, _WIFEXITED=os.WIFEXITED,
1580 _WEXITSTATUS=os.WEXITSTATUS):
1581 """All callers to this function MUST hold self._waitpid_lock."""
1582 # This method is called (indirectly) by __del__, so it cannot
1583 # refer to anything outside of its local scope."""
1584 if _WIFSIGNALED(sts):
1585 self.returncode = -_WTERMSIG(sts)
1586 elif _WIFEXITED(sts):
1587 self.returncode = _WEXITSTATUS(sts)
1588 else:
1589 # Should never happen
1590 raise RuntimeError("Unknown child exit status!")
1591
1592
1593 def _internal_poll(self, _deadstate=None, _waitpid=os.waitpid,
1594 _WNOHANG=os.WNOHANG, _os_error=os.error, _ECHILD=errno.ECHILD):
1595 """Check if child process has terminated. Returns returncode
1596 attribute.
1597
1598 This method is called by __del__, so it cannot reference anything
1599 outside of the local scope (nor can any methods it calls).
1600
1601 """
1602 if self.returncode is None:
1603 if not self._waitpid_lock.acquire(False):
1604 # Something else is busy calling waitpid. Don't allow two
1605 # at once. We know nothing yet.
1606 return None
1607 try:
1608 try:
1609 if self.returncode is not None:
1610 return self.returncode # Another thread waited.
1611 pid, sts = _waitpid(self.pid, _WNOHANG)
1612 if pid == self.pid:
1613 self._handle_exitstatus(sts)
1614 except _os_error, e:
1615 if _deadstate is not None:
1616 self.returncode = _deadstate
1617 elif e.errno == _ECHILD:
1618 # This happens if SIGCLD is set to be ignored or
1619 # waiting for child processes has otherwise been
1620 # disabled for our process. This child is dead, we
1621 # can't get the status.
1622 # http://bugs.python.org/issue15756
1623 self.returncode = 0
1624 finally:
1625 self._waitpid_lock.release()
1626 return self.returncode
1627
1628
1629 def _try_wait(self, wait_flags):
1630 """All callers to this function MUST hold self._waitpid_lock."""
1631 try:
1632 (pid, sts) = _eintr_retry_call(os.waitpid, self.pid, wait_flags)
1633 except OSError, e:
1634 if e.errno != errno.ECHILD:
1635 raise
1636 # This happens if SIGCLD is set to be ignored or waiting
1637 # for child processes has otherwise been disabled for our
1638 # process. This child is dead, we can't get the status.
1639 pid = self.pid
1640 sts = 0
1641 return (pid, sts)
1642
1643
1644 def wait(self, timeout=None, endtime=None):
1645 """Wait for child process to terminate. Returns returncode
1646 attribute."""
1647 if self.returncode is not None:
1648 return self.returncode
1649
1650 # endtime is preferred to timeout. timeout is only used for
1651 # printing.
1652 if endtime is not None or timeout is not None:
1653 if endtime is None:
1654 endtime = time.time() + timeout
1655 elif timeout is None:
1656 timeout = self._remaining_time(endtime)
1657
1658 if endtime is not None:
1659 # Enter a busy loop if we have a timeout. This busy loop was
1660 # cribbed from Lib/threading.py in Thread.wait() at r71065.
1661 delay = 0.0005 # 500 us -> initial delay of 1 ms
1662 while True:
1663 if self._waitpid_lock.acquire(False):
1664 try:
1665 if self.returncode is not None:
1666 break # Another thread waited.
1667 (pid, sts) = self._try_wait(os.WNOHANG)
1668 assert pid == self.pid or pid == 0
1669 if pid == self.pid:
1670 self._handle_exitstatus(sts)
1671 break
1672 finally:
1673 self._waitpid_lock.release()
1674 remaining = self._remaining_time(endtime)
1675 if remaining <= 0:
1676 raise TimeoutExpired(self.args, timeout)
1677 delay = min(delay * 2, remaining, .05)
1678 time.sleep(delay)
1679 else:
1680 while self.returncode is None:
1681 self._waitpid_lock.acquire()
1682 try:
1683 if self.returncode is not None:
1684 break # Another thread waited.
1685 (pid, sts) = self._try_wait(0)
1686 # Check the pid and loop as waitpid has been known to
1687 # return 0 even without WNOHANG in odd situations.
1688 # http://bugs.python.org/issue14396.
1689 if pid == self.pid:
1690 self._handle_exitstatus(sts)
1691 finally:
1692 self._waitpid_lock.release()
1693 return self.returncode
1694
1695
1696 def _communicate(self, input, endtime, orig_timeout):
1697 if self.stdin and not self._communication_started:
1698 # Flush stdio buffer. This might block, if the user has
1699 # been writing to .stdin in an uncontrolled fashion.
1700 self.stdin.flush()
1701 if not input:
1702 self.stdin.close()
1703
1704 if _has_poll:
1705 stdout, stderr = self._communicate_with_poll(input, endtime,
1706 orig_timeout)
1707 else:
1708 stdout, stderr = self._communicate_with_select(input, endtime,
1709 orig_timeout)
1710
1711 self.wait(timeout=self._remaining_time(endtime))
1712
1713 # All data exchanged. Translate lists into strings.
1714 if stdout is not None:
1715 stdout = ''.join(stdout)
1716 if stderr is not None:
1717 stderr = ''.join(stderr)
1718
1719 # Translate newlines, if requested. We cannot let the file
1720 # object do the translation: It is based on stdio, which is
1721 # impossible to combine with select (unless forcing no
1722 # buffering).
1723 if self.universal_newlines and hasattr(file, 'newlines'):
1724 if stdout:
1725 stdout = self._translate_newlines(stdout)
1726 if stderr:
1727 stderr = self._translate_newlines(stderr)
1728
1729 return (stdout, stderr)
1730
1731
1732 def _communicate_with_poll(self, input, endtime, orig_timeout):
1733 stdout = None # Return
1734 stderr = None # Return
1735
1736 if not self._communication_started:
1737 self._fd2file = {}
1738
1739 poller = select.poll()
1740 def register_and_append(file_obj, eventmask):
1741 poller.register(file_obj.fileno(), eventmask)
1742 self._fd2file[file_obj.fileno()] = file_obj
1743
1744 def close_unregister_and_remove(fd):
1745 poller.unregister(fd)
1746 self._fd2file[fd].close()
1747 self._fd2file.pop(fd)
1748
1749 if self.stdin and input:
1750 register_and_append(self.stdin, select.POLLOUT)
1751
1752 # Only create this mapping if we haven't already.
1753 if not self._communication_started:
1754 self._fd2output = {}
1755 if self.stdout:
1756 self._fd2output[self.stdout.fileno()] = []
1757 if self.stderr:
1758 self._fd2output[self.stderr.fileno()] = []
1759
1760 select_POLLIN_POLLPRI = select.POLLIN | select.POLLPRI
1761 if self.stdout:
1762 register_and_append(self.stdout, select_POLLIN_POLLPRI)
1763 stdout = self._fd2output[self.stdout.fileno()]
1764 if self.stderr:
1765 register_and_append(self.stderr, select_POLLIN_POLLPRI)
1766 stderr = self._fd2output[self.stderr.fileno()]
1767
1768 # Save the input here so that if we time out while communicating,
1769 # we can continue sending input if we retry.
1770 if self.stdin and self._input is None:
1771 self._input_offset = 0
1772 self._input = input
1773 if self.universal_newlines and isinstance(self._input, unicode):
1774 self._input = self._input.encode(
1775 self.stdin.encoding or sys.getdefaultencoding())
1776
1777 while self._fd2file:
1778 try:
1779 ready = poller.poll(self._remaining_time(endtime))
1780 except select.error, e:
1781 if e.args[0] == errno.EINTR:
1782 continue
1783 raise
1784 self._check_timeout(endtime, orig_timeout)
1785
1786 for fd, mode in ready:
1787 if mode & select.POLLOUT:
1788 chunk = self._input[self._input_offset :
1789 self._input_offset + _PIPE_BUF]
1790 self._input_offset += os.write(fd, chunk)
1791 if self._input_offset >= len(self._input):
1792 close_unregister_and_remove(fd)
1793 elif mode & select_POLLIN_POLLPRI:
1794 data = os.read(fd, 4096)
1795 if not data:
1796 close_unregister_and_remove(fd)
1797 self._fd2output[fd].append(data)
1798 else:
1799 # Ignore hang up or errors.
1800 close_unregister_and_remove(fd)
1801
1802 return (stdout, stderr)
1803
1804
1805 def _communicate_with_select(self, input, endtime, orig_timeout):
1806 if not self._communication_started:
1807 self._read_set = []
1808 self._write_set = []
1809 if self.stdin and input:
1810 self._write_set.append(self.stdin)
1811 if self.stdout:
1812 self._read_set.append(self.stdout)
1813 if self.stderr:
1814 self._read_set.append(self.stderr)
1815
1816 if self.stdin and self._input is None:
1817 self._input_offset = 0
1818 self._input = input
1819 if self.universal_newlines and isinstance(self._input, unicode):
1820 self._input = self._input.encode(
1821 self.stdin.encoding or sys.getdefaultencoding())
1822
1823 stdout = None # Return
1824 stderr = None # Return
1825
1826 if self.stdout:
1827 if not self._communication_started:
1828 self._stdout_buff = []
1829 stdout = self._stdout_buff
1830 if self.stderr:
1831 if not self._communication_started:
1832 self._stderr_buff = []
1833 stderr = self._stderr_buff
1834
1835 while self._read_set or self._write_set:
1836 try:
1837 (rlist, wlist, xlist) = \
1838 select.select(self._read_set, self._write_set, [],
1839 self._remaining_time(endtime))
1840 except select.error, e:
1841 if e.args[0] == errno.EINTR:
1842 continue
1843 raise
1844
1845 # According to the docs, returning three empty lists indicates
1846 # that the timeout expired.
1847 if not (rlist or wlist or xlist):
1848 raise TimeoutExpired(self.args, orig_timeout)
1849 # We also check what time it is ourselves for good measure.
1850 self._check_timeout(endtime, orig_timeout)
1851
1852 if self.stdin in wlist:
1853 chunk = self._input[self._input_offset :
1854 self._input_offset + _PIPE_BUF]
1855 bytes_written = os.write(self.stdin.fileno(), chunk)
1856 self._input_offset += bytes_written
1857 if self._input_offset >= len(self._input):
1858 self.stdin.close()
1859 self._write_set.remove(self.stdin)
1860
1861 if self.stdout in rlist:
1862 data = os.read(self.stdout.fileno(), 1024)
1863 if data == "":
1864 self.stdout.close()
1865 self._read_set.remove(self.stdout)
1866 stdout.append(data)
1867
1868 if self.stderr in rlist:
1869 data = os.read(self.stderr.fileno(), 1024)
1870 if data == "":
1871 self.stderr.close()
1872 self._read_set.remove(self.stderr)
1873 stderr.append(data)
1874
1875 return (stdout, stderr)
1876
1877
1878 def send_signal(self, sig):
1879 """Send a signal to the process
1880 """
1881 os.kill(self.pid, sig)
1882
1883 def terminate(self):
1884 """Terminate the process with SIGTERM
1885 """
1886 self.send_signal(signal.SIGTERM)
1887
1888 def kill(self):
1889 """Kill the process with SIGKILL
1890 """
1891 self.send_signal(signal.SIGKILL)
OLDNEW
« no previous file with comments | « tools/telemetry/third_party/subprocess32/setup.py ('k') | tools/telemetry/third_party/subprocess32/test_subprocess32.py » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698