|
|
Created:
3 years, 9 months ago by shivanisha Modified:
3 years, 6 months ago CC:
cbentzel+watch_chromium.org, chromium-reviews, gavinp+disk_chromium.org, net-reviews_chromium.org Target Ref:
refs/heads/master Project:
chromium Visibility:
Public. |
DescriptionThis CL is a precursor to allowing shared writing to fix cache lock.
This CL allows transactions to continue to their validation phase even when another
transaction is the active reader/writer. After the validation phase, if its a match
the transaction might wait till the response is written to the cache by the active
writer. If its not a match the transaction will doom the entry and go to the
network. In a subsequent CL, the not matching case will create a new entry as well.
BUG=472740, 715913, 715974, 715920, 715911, 713348
Review-Url: https://codereview.chromium.org/2721933002
Cr-Original-Commit-Position: refs/heads/master@{#467426}
Committed: https://chromium.googlesource.com/chromium/src/+/1e2e347f957ef889aaee527bb757849f76e8a808
Review-Url: https://codereview.chromium.org/2721933002
Cr-Commit-Position: refs/heads/master@{#479204}
Committed: https://chromium.googlesource.com/chromium/src/+/8061c420676998bda77caa74581ea8061860f438
Patch Set 1 : Initial patch #
Total comments: 75
Patch Set 2 : Feedback addressed, more tests and refactoring #
Total comments: 15
Patch Set 3 : Rough patch to discuss if ActiveEntry would be better designed with a state machine #
Total comments: 1
Patch Set 4 : ActiveEntry state machine (testing in progress, uploading for discussion) #Patch Set 5 : Simplified ActiveEntry's state transitions #
Total comments: 42
Patch Set 6 : Feedback addresses + rebased till refs/heads/master@{#458081} #Patch Set 7 : HttpCache::Transaction feedback addressed #Patch Set 8 : DoLoop modified #Patch Set 9 : Use TransitionToState (Rebased till refs/heads/master@{#459057}) #
Total comments: 7
Patch Set 10 : Fixed DoneWithEntry for headers_transaction #
Total comments: 9
Patch Set 11 : Fixed HttpCache.StopCachingWithAuthDeletesEntry to not invoke Read after receiving a 401 #Patch Set 12 : Josh's latest feedback addressed. #
Total comments: 1
Patch Set 13 : HttpCache::IsCancelResponseBody added. #
Total comments: 36
Patch Set 14 : Josh's feedback on HttpCache::Transaction addressed #
Total comments: 16
Patch Set 15 : Feedback addressed (based on refs/heads/master@{#459057}) #
Total comments: 23
Patch Set 16 : Feedback addressed. #
Total comments: 2
Patch Set 17 : Tests. #Patch Set 18 : Josh's latest feedback addressed #
Total comments: 9
Patch Set 19 : Randy's feedback addressed. #Patch Set 20 : Rebased till refs/heads/master@{#462134} #
Total comments: 48
Patch Set 21 : Pause and restart network transaction framework, tests based on that. #Patch Set 22 : Feedback addressed #Patch Set 23 : Rebased with refs/heads/master@{#463004} #
Total comments: 19
Patch Set 24 : Remove loop from ProcessDoneHeadersQueue #Patch Set 25 : Feedback addressed #
Total comments: 2
Patch Set 26 : Moved DeferNetworkStart, fixed a bot failure #
Total comments: 14
Patch Set 27 : Writer returns synchronously from DoneWithResponseHeaders #
Total comments: 4
Patch Set 28 : Fixed data race #
Total comments: 15
Patch Set 29 : Rebased + test-only changes for fixing the new Push Cache Lookup failing unit tests #
Total comments: 46
Patch Set 30 : Feedback addressed #Patch Set 31 : Feedback addressed #
Total comments: 8
Patch Set 32 : Feedback addressed #
Total comments: 2
Patch Set 33 : Comments simplified #
Total comments: 2
Patch Set 34 : Cached HEAD request handling fixed #Patch Set 35 : Fixed partial transactions handling in ActiveEntry #
Total comments: 3
Patch Set 36 : MetadataWriter to initialize request_info_ before transaction_ #Patch Set 37 : Fixing HttpTransaction::Start()'s comment for |*request_info| for range requests. #Patch Set 38 : Adding log to understand the webkit_tests failure. #Patch Set 39 : Fixing resetting the cache_entry_status_. Should fix the webkit_tests failing in linux_trusty_blinkā¦ #
Total comments: 2
Patch Set 40 : Made IsTransactionWritingIncomplete clearer for range requests #Patch Set 41 : Simplified HttpTransaction::Start API contract for request info #
Total comments: 7
Patch Set 42 : Range requests tests + refactoring DoneWithEntry #Patch Set 43 : Convert DCHECK -> CHECK for important invariants. #
Total comments: 6
Patch Set 44 : restart_info_.cache_entry_status to be set unconditionally #Patch Set 45 : TODOs added for CHECKs and Transaction::Start #
Total comments: 4
Patch Set 46 : DoneReadingFromEntry replaced with DoneWithEntry #
Total comments: 9
Patch Set 47 : Rebased with refs/heads/master@{#475007} #
Total comments: 2
Patch Set 48 : rdsmith feedback. #
Total comments: 4
Patch Set 49 : jkarlin feedback + remove a dcheck based on a crash #
Total comments: 3
Patch Set 50 : IsTransactionExclusivelyWriting simplified and renamed to HasDependentTransactions #
Total comments: 8
Patch Set 51 : Truncation and StopCaching changes. #
Total comments: 34
Patch Set 52 : Feedback addressed #Patch Set 53 : CanTransactionWriteResponseHeaders to take care of writer. #Patch Set 54 : nit addressed #Messages
Total messages: 286 (155 generated)
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Description was changed from ========== Draft CL (not for review) BUG= ========== to ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. More tests are coming. BUG=472740 ==========
Description was changed from ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. More tests are coming. BUG=472740 ========== to ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. More tests are coming. BUG=472740 ==========
Patchset #1 (id:1) has been deleted
Patchset #1 (id:20001) has been deleted
Patchset #1 (id:40001) has been deleted
shivanisha@chromium.org changed reviewers: + jkarlin@chromium.org, rdsmith@chromium.org
PTAL, Thanks! https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_uni... File net/http/http_cache_unittest.cc (left): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_uni... net/http/http_cache_unittest.cc:6849: // Tests that if a metadata writer transaction hits cache lock timeout, it will This is a test from an unrelated CL https://codereview.chromium.org/2671793002. Removed it since this test doesn't really test what it is supposed to. Reason being that MetadataWriter does not use CreateTransaction but calls "new" directly and thus does not fill the bypass_lock_for_test_ field.
Exciting. Here is a first round of comments. Something that makes this hard is that the parallel validating entry think it's a writer, so all of the HttpCache methods have to carefully differentiate between the actual writer and the parallel validators. I wonder if this could be made cleaner somehow. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:700: if (!entry_alive || entry != entry_alive) return entry_alive && entry_alive == entry; https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:845: bool validated_queue, We discussed this F2F, but can you remove validated_queue and separate that logic into a separate function? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:846: bool new_transaction) { is new_transaction ever set to true? I don't see it from any callers from HttpCache or HttpCacheTransaction https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:905: entry->readers.insert(transaction); We're adding it to readers, but I don't think the transactions mode is set to READ yet is it? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:926: auto i = std::find(entry->validated_queue.begin(), There are a few linear searches in this new code. Not something to fix yet but perhaps make a note of it. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:241: class WorkItem; This isn't alphabetical. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:251: class ActiveEntry { If it's a class the members need to be private and have a _ suffix. This is still reasonable as a struct. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:273: TransactionList pending_queue; pending_queue seems a bit vague now. Perhaps add_to_entry_queue? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:334: // Returns true if given entry is still active. What does it mean for an entry to be active? It's called "ActiveEntry" ;) Perhaps Doomed() or IsDoomed? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:371: // method returns ERR_CACHE_RACE to signal the transaction that it cannot be I don't think this actually does return ERR_CACHE_RACE. Can you remove that part of the comment? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:373: // one (in this case, the entry is no longer valid). validation_done should |validation_done|. We wrap variable names in ||'s. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:375: // phase, new_transaction should be passed as true if this transaction is a same with |new_transaction| https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:379: bool validation_done = false, nit: s/validation_done/validation_needed/ https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:380: bool new_transaction = false); nit: is_new_transaction to make it clear that it's a bool and not a transaction type https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:380: bool new_transaction = false); This method has few enough existing callers that I don't think we should use default values and instead update the callers. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:388: int OnValidationMatch(ActiveEntry* entry, In keeping with the naming scheme, perhaps DoneWithValidation? You could combine these two functions into one. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:458: // IO callbacks with ERR_CACHE_RACE. Is this actually a cache race though? Seems like a misuse of the flag. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:726: if (!reading_ && rv == OK) { How many ways can we get here? Can we just call 'return WaitBeforeRead()' which either returns OK or sets the next state to STATE_WAIT_BEFORE_READ from the necessary locations? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:732: read_buf_ = NULL; // Release the buffer before invoking the callback. s/NULL/nullptr/ https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1792: if (write_this_response_) { Seems clearer to call entry_->writer == this && request_->method != "HEAD" here https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1793: cache_->OnValidationMatch(entry_, this, true); OnValidationMatch(entry_, this, true /* write_this_response */); https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1811: return cache_->OnValidationMatch(entry_, this, false); OnValidationMatch(entry_, this, false /* write_this_response */); https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1832: mode_ = READ; I'd have thought that the existing code would have already marked this transaction as a reader if the validation matched. It looks like SetupEntryForRead does. But perhaps the 304 path does not? Should it? If so, then we can just DCHECK the mode here. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1834: ResetNetworkTransaction(); Likewise, shouldn't this be taken care of already? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:2671: if (cache_->ConvertWriterToReader(entry_, this)) I see, this is why we had to change mode_ = READ in DoWaitBeforeReadComplete. Perhaps we can set mode_ = READ reagardless? Also, ConvertWriterToReader now handles cases where the current transaction isn't actually a writer. Perhaps we should be calling a different function if it's not the writer? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.h:472: bool write_this_response_; This variable seems unnecessary, it's only read in one place which could just as easily check if it's the entry's writers.
https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:241: class WorkItem; On 2017/03/07 16:45:41, jkarlin wrote: > This isn't alphabetical. I actually don't know that they need to be alphabetical, but it's nicer if they are :)
Shivani: The comments below don't really even count as a first pass, as a fair amount of information has drained out of my head since the last time I looked at this code and I need to fault it back in. A couple of high level questions and then I'll dive back into the review: * What are the transition pathways for transactions in HttpCache::ActiveEntry? I'm guessing it's pending_queue -> validating_transaction -> validated_queue -> {readers, writer}; is that right? (As noted below, I think this would be good to document this in the comments in the structure, but I also need to be solidly on top of this to do the review, so I'm asking top level as well.) * It seems like the point of using PostTask() for ProcessPendingQueue() and ProcessParallelValidation(), which is to give the readers queue a chance to completely clear in case a writer needs to be put into the entry by AddEntryToTransacation. But that doesn't make sense, because checking to see if anything in the pending queue can be transitioned to writer/readers based on its mode and the state of writer/readers seems like a fairly quick check, so why not just do it on each call to ProcessPendingQueue (or parallel validation)? Thanks for this/sorry for the educational effort involved (though arguably it helps to figure out where code documentation is useful and where it isn't). https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc File net/http/http_cache.cc (left): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:699: DCHECK(!entry->will_process_pending_queue); Why? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:634: DCHECK(entry_ptr->writer || !entry_ptr->readers.empty() || Do you know what the purpose of this DCHECK is? It seems to be saying "You can only doom an entry if some transaction is using it", and I'm curious if that's true and if so what the logic is behind that assertion. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:868: // become the writer if there is no active writer at this time. Is there a reason for it to take up the writer slot before validation is complete? I don't think we want to rule out another transaction coming in later but finishing validation first and starting to write. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:890: ProcessPendingQueue(entry); nit, suggestion: It feels like this routine might be clearer if (e.g.) assignment in pending queue in these last several lines was unconditional, and then in ProcessPendingQueue situations like the lack of writer and a non-empty pending_queue were detected, and ProcessPendingQueue pushed the state along. If you agree with this suggestion, you might want to scan the other functions to see if it applies; having a single or a small number of functions with state advancing logic can make that state advancing logic much easier to understand. (I suspect it's too much of a refactor, but if you want to and think it would be a good idea, I'd supporting converting all such routines into an AdvanceState() routine and really centralize the state logic.) https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:929: entry->validated_queue.erase(i); Can't it be in the validated queue and be the writer? In that situation, won't the writer remain set to the transaction? https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:945: if (transaction == entry->writer || Suggestion: I'd value a comment here, parallel to the ones above. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:970: DCHECK_EQ(transaction, entry->writer); Suggestion: Hoist this up tot he top of the routine and make it DCHECK(transaction == entry->writer || transaction == entry->validating_transacation). That way it acts as a contract check as well as documentation at the beginning of the routine as to the expected environment. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:971: entry->writer = nullptr; Suggestion: If you take the above suggestion, you could hoist the "if (transaction == entry->writer) entry->writer = nullptr" up to the beginning of the routine and eliminate this line. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:1208: void HttpCache::DoomEntryRestartPendingTransactions(ActiveEntry* entry) { This function and the next one look simple and are only called from one place. What's your thinking about having them as separate functions rather than hoisting them into their callers? (This may be my focussing on API cleanliness at the expense of other forms, but if so I'd like to understand the other advantages I'm missing.) https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:272: // Transactions waiting to be added to entry. nit: Thank you for the extra comments; they help! Would you be willing to expand them with the queue/element transition pathways? I'm guessing that transactions move from pending_queue -> validating_transaction -> validated_queue -> {writer/readers}, but I'm not sure and it would be wonderful to have that here. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:417: // transaction was added to readers. It's not clear to me what this documentation means; it's sorta "Do this thing. If you don't do this thing, return false". Which makes it sound like false is returned in an error case. But if I look at the code, it's more that this function moves a transaction out of a state in which its either the writer or the validating transaction into whatever the right next state is (reader or pending), and we return true/false to indicate what state it's actually moved into. Could we change the name and documentation to make this clearer? Looking at the calling locations, I actually think "DoneValidatingEntry" may make sense as a name, with the doc being "Signals that a transaction's cache update (writing/validation) is done and it should be moved into its next state. Returns true if the entry is now a reader". This is a specific instance of a general issue that I care about a lot. I really want header comments to make sense from the viewpoint of the consumer *without* the consumer having to look at the code of the producing class. The producing class should implement some abstraction that's higher level than its member variables, and should talk in the comments on its methods in the language of that higher layer abstraction. If that isn't true, that's a layering violation. Sometimes that's the best thing (sometimes avoiding layering violations require way too much effort to make it worthwhile) but I'm usually going to treat comments that leak information about the class implementation the same way I'd treat code, i.e. with a "Is there a better way to do this?" question. As a secondary point, I do wonder why the HttpCache::Transaction can't switch to read mode without the transaction actually being in the readers set in the HttpCache::ActiveEntry. When we later finish up with a writer transaction, won't we check the modes of the transactions in pending_transactions before we set a new writer? So shouldn't it be safe to just unilaterally change the mode to READ in the transaction even if the cache didn't transition it into the readers queue? This would allow getting rid of the bool completely.
While I am working on the review comments, responding to the high level comments here: On 2017/03/07 at 22:54:30, rdsmith wrote: > Shivani: The comments below don't really even count as a first pass, as a fair amount of information has drained out of my head since the last time I looked at this code and I need to fault it back in. A couple of high level questions and then I'll dive back into the review: > > * What are the transition pathways for transactions in HttpCache::ActiveEntry? I'm guessing it's pending_queue -> validating_transaction -> validated_queue -> {readers, writer}; is that right? (As noted below, I think this would be good to document this in the comments in the structure, but I also need to be solidly on top of this to do the review, so I'm asking top level as well.) > Will surely document these in the code. write mode: [pending_queue]-> validating_transaction & [writer] -> [validated_queue] -> writer [pending_queue]-> validating_transaction -> [validated_queue] -> writer (if the original writer could not write to the cache because of failure/cancellation) [pending_queue]-> validating_transaction -> [validated_queue] -> reader (once the data is written to the cache by the writer) read-only: [pending_queue]-> readers (once the data is written to the cache by the writer) > * It seems like the point of using PostTask() for ProcessPendingQueue() and ProcessParallelValidation(), which is to give the readers queue a chance to completely clear in case a writer needs to be put into the entry by AddEntryToTransacation. But that doesn't make sense, because checking to see if anything in the pending queue can be transitioned to writer/readers based on its mode and the state of writer/readers seems like a fairly quick check, so why not just do it on each call to ProcessPendingQueue (or parallel validation)? The only advantage I see of posting it as a task is to not have to go to another transaction's (possibly more than 1 if its a reader) state machine(s) via its IO callback while we are in the control flow of another transaction. It seems simpler to have one transaction's state machine in one call stack. > > Thanks for this/sorry for the educational effort involved (though arguably it helps to figure out where code documentation is useful and where it isn't). > -
On 2017/03/08 16:47:46, shivanisha wrote: > While I am working on the review comments, responding to the high level comments > here: > > On 2017/03/07 at 22:54:30, rdsmith wrote: > > Shivani: The comments below don't really even count as a first pass, as a fair > amount of information has drained out of my head since the last time I looked at > this code and I need to fault it back in. A couple of high level questions and > then I'll dive back into the review: > > > > * What are the transition pathways for transactions in HttpCache::ActiveEntry? > I'm guessing it's pending_queue -> validating_transaction -> validated_queue -> > {readers, writer}; is that right? (As noted below, I think this would be good > to document this in the comments in the structure, but I also need to be solidly > on top of this to do the review, so I'm asking top level as well.) > > > Will surely document these in the code. > write mode: > [pending_queue]-> validating_transaction & [writer] -> [validated_queue] -> > writer > [pending_queue]-> validating_transaction -> [validated_queue] -> writer (if the > original writer could not write to the cache because of failure/cancellation) Why validating_transactions & [writer]? I would imagine the code would be simpler to understand if a transaction was in only one of the buckets at a time. > [pending_queue]-> validating_transaction -> [validated_queue] -> reader (once > the data is written to the cache by the writer) > > read-only: > [pending_queue]-> readers (once the data is written to the cache by the writer) > > > * It seems like the point of using PostTask() for ProcessPendingQueue() and > ProcessParallelValidation(), which is to give the readers queue a chance to > completely clear in case a writer needs to be put into the entry by > AddEntryToTransacation. But that doesn't make sense, because checking to see > if anything in the pending queue can be transitioned to writer/readers based on > its mode and the state of writer/readers seems like a fairly quick check, so why > not just do it on each call to ProcessPendingQueue (or parallel validation)? > > The only advantage I see of posting it as a task is to not have to go to another > transaction's (possibly more than 1 if its a reader) state machine(s) via its IO > callback while we are in the control flow of another transaction. It seems > simpler to have one transaction's state machine in one call stack. Simpler and avoids re-entrancy. Ok, that makes sense. (Though I may want to document that as well.) Ok, I'll dive back into the review now. Thanks for the explanations! > > > > > Thanks for this/sorry for the educational effort involved (though arguably it > helps to figure out where code documentation is useful and where it isn't). > > > > -
On 2017/03/08 at 18:17:10, rdsmith wrote: > On 2017/03/08 16:47:46, shivanisha wrote: > > While I am working on the review comments, responding to the high level comments > > here: > > > > On 2017/03/07 at 22:54:30, rdsmith wrote: > > > Shivani: The comments below don't really even count as a first pass, as a fair > > amount of information has drained out of my head since the last time I looked at > > this code and I need to fault it back in. A couple of high level questions and > > then I'll dive back into the review: > > > > > > * What are the transition pathways for transactions in HttpCache::ActiveEntry? > > I'm guessing it's pending_queue -> validating_transaction -> validated_queue -> > > {readers, writer}; is that right? (As noted below, I think this would be good > > to document this in the comments in the structure, but I also need to be solidly > > on top of this to do the review, so I'm asking top level as well.) > > > > > Will surely document these in the code. > > write mode: > > [pending_queue]-> validating_transaction & [writer] -> [validated_queue] -> > > writer > > [pending_queue]-> validating_transaction -> [validated_queue] -> writer (if the > > original writer could not write to the cache because of failure/cancellation) > > Why validating_transactions & [writer]? I would imagine the code would be simpler to understand if a transaction was in only one of the buckets at a time. The writer slot works as a lock (in the existing code and in this CL). Presence of the writer decides whether a new reader can be added or not. Similarly, since we only allow one validating transaction at a time, validating_transaction acts as a lock that no other transaction can start validating. > > > [pending_queue]-> validating_transaction -> [validated_queue] -> reader (once > > the data is written to the cache by the writer) > > > > read-only: > > [pending_queue]-> readers (once the data is written to the cache by the writer) > > > > > * It seems like the point of using PostTask() for ProcessPendingQueue() and > > ProcessParallelValidation(), which is to give the readers queue a chance to > > completely clear in case a writer needs to be put into the entry by > > AddEntryToTransacation. But that doesn't make sense, because checking to see > > if anything in the pending queue can be transitioned to writer/readers based on > > its mode and the state of writer/readers seems like a fairly quick check, so why > > not just do it on each call to ProcessPendingQueue (or parallel validation)? > > > > The only advantage I see of posting it as a task is to not have to go to another > > transaction's (possibly more than 1 if its a reader) state machine(s) via its IO > > callback while we are in the control flow of another transaction. It seems > > simpler to have one transaction's state machine in one call stack. > > Simpler and avoids re-entrancy. Ok, that makes sense. (Though I may want to document that as well.) Sure, will document this. > > Ok, I'll dive back into the review now. Thanks for the explanations! > > > > > > > > > Thanks for this/sorry for the educational effort involved (though arguably it > > helps to figure out where code documentation is useful and where it isn't). > > > > > All clarification comments are most welcome. Like you mention, they do also help to figure out where documentation is lacking. More documentation in this part of the code will really help! > > -
On 2017/03/08 at 18:24:54, shivanisha wrote: > On 2017/03/08 at 18:17:10, rdsmith wrote: > > On 2017/03/08 16:47:46, shivanisha wrote: > > > While I am working on the review comments, responding to the high level comments > > > here: > > > > > > On 2017/03/07 at 22:54:30, rdsmith wrote: > > > > Shivani: The comments below don't really even count as a first pass, as a fair > > > amount of information has drained out of my head since the last time I looked at > > > this code and I need to fault it back in. A couple of high level questions and > > > then I'll dive back into the review: > > > > > > > > * What are the transition pathways for transactions in HttpCache::ActiveEntry? > > > I'm guessing it's pending_queue -> validating_transaction -> validated_queue -> > > > {readers, writer}; is that right? (As noted below, I think this would be good > > > to document this in the comments in the structure, but I also need to be solidly > > > on top of this to do the review, so I'm asking top level as well.) > > > > > > > Will surely document these in the code. > > > write mode: > > > [pending_queue]-> validating_transaction & [writer] -> [validated_queue] -> > > > writer > > > [pending_queue]-> validating_transaction -> [validated_queue] -> writer (if the > > > original writer could not write to the cache because of failure/cancellation) > > > > Why validating_transactions & [writer]? I would imagine the code would be simpler to understand if a transaction was in only one of the buckets at a time. > > The writer slot works as a lock (in the existing code and in this CL). Presence of the writer decides whether a new reader can be added or not. Similarly, since we only allow one validating transaction at a time, validating_transaction acts as a lock that no other transaction can start validating. > Here is a better/simplified representation of the transitions: Write mode transactions: [pending_queue]-> validating_transaction & writer -> writer [pending_queue]-> validating_transaction -> validated_queue -> reader (once the data is written to the cache by another writer) [pending_queue]-> validating_transaction -> writer (if the original writer could not write to the cache because of failure/cancellation) Read only transactions: [pending_queue]-> readers (once the data is written to the cache by the writer) > > > > > [pending_queue]-> validating_transaction -> [validated_queue] -> reader (once > > > the data is written to the cache by the writer) > > > > > > read-only: > > > [pending_queue]-> readers (once the data is written to the cache by the writer) > > > > > > > * It seems like the point of using PostTask() for ProcessPendingQueue() and > > > ProcessParallelValidation(), which is to give the readers queue a chance to > > > completely clear in case a writer needs to be put into the entry by > > > AddEntryToTransacation. But that doesn't make sense, because checking to see > > > if anything in the pending queue can be transitioned to writer/readers based on > > > its mode and the state of writer/readers seems like a fairly quick check, so why > > > not just do it on each call to ProcessPendingQueue (or parallel validation)? > > > > > > The only advantage I see of posting it as a task is to not have to go to another > > > transaction's (possibly more than 1 if its a reader) state machine(s) via its IO > > > callback while we are in the control flow of another transaction. It seems > > > simpler to have one transaction's state machine in one call stack. > > > > Simpler and avoids re-entrancy. Ok, that makes sense. (Though I may want to document that as well.) > > Sure, will document this. > > > > > Ok, I'll dive back into the review now. Thanks for the explanations! > > > > > > > > > > > > > Thanks for this/sorry for the educational effort involved (though arguably it > > > helps to figure out where code documentation is useful and where it isn't). > > > > > > > > All clarification comments are most welcome. Like you mention, they do also help to figure out where documentation is lacking. More documentation in this part of the code will really help! > > > -
Shivani: A high level design suggestion I want to run by you. It might not work (== moderately simplify the code) but if it does and you like it I'd rather do the review afterwards instead of before :-}. The pattern I find myself pulling towards for the HttpCache additions in this CL is something like: * There is a state machine that advances an ActiveEntry * There is a single function that executes this state machine. * All the information needed to figure out how to advance the ActiveEntry is in the ActiveEntry or in the transactions in the ActiveEntry. This would mean that OnProcessPendingQueue & OnProcessParallelValidation could be combined, and I believe that state transitions in other parts of the code could be simplified. (e.g. if a transaction was done being validated you'd put it on the validated queue and invoke the state advancement machinery--no testing of writer or readers to figure out what happens next.) And it would mean evaluating the state machine would be much simpler, because there would just be one section of the code to read. I think this might help with AddTransactionToEntry and its arguments as well; specifically, this suggestion would mean that validated_queue would be (in some fashion) embodied on the transaction and hence wouldn't need to be an argument. If you see problems with this suggestion, let me know, since they'll almost certainly be design issues I'll need to take into account in the review anyway :-}. I'm going to continue on and review the cache transaction, but I'll hold off on further review of HttpCache until I hear back from you on this. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:272: // Transactions waiting to be added to entry. On 2017/03/07 22:54:30, Randy Smith (Not in Mondays) wrote: > nit: Thank you for the extra comments; they help! Would you be willing to > expand them with the queue/element transition pathways? I'm guessing that > transactions move from pending_queue -> validating_transaction -> > validated_queue -> {writer/readers}, but I'm not sure and it would be wonderful > to have that here. > minor additional nit/suggestion: Ordering the members in the basic order a transaction goes through them might be useful. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:380: bool new_transaction = false); On 2017/03/07 16:45:41, jkarlin wrote: > This method has few enough existing callers that I don't think we should use > default values and instead update the callers. The style guide generally recommends against default arguments (which they equivalence to overloaded functions). See https://google.github.io/styleguide/cppguide.html#Function_Overloading and the following. Can you give a sense as to why this should be an exception? It's often better to have separate functions with descriptive names, since that makes it easier for people reading at the call site to understand exactly what's being done.
This patch addresses most feedback except Randy's most recent high level design suggestion (thinking over it). A lot of functions have changed as part of refactoring/simplification. Some more tests are also added. PTAL, thanks! https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc File net/http/http_cache.cc (left): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:699: DCHECK(!entry->will_process_pending_queue); On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > Why? Now that we have 2 types of tasks posted (OnProcessWaitingTransactions and OnProcessParallelValidation), one of them can destroy the entry if there are no transactions while the other one may be posted, thus removed this assert and also added a IsEntryAlive check in both tasks. (Note the names of functions here correspond to the new patch) https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:634: DCHECK(entry_ptr->writer || !entry_ptr->readers.empty() || On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > Do you know what the purpose of this DCHECK is? It seems to be saying "You can only doom an entry if some transaction is using it", and I'm curious if that's true and if so what the logic is behind that assertion. DoomEntry's consumer is the HttpCache::Transaction (either directly or via DoomActiveEntry or DoomMainEntryForUrl). There are various places where HCT dooms the entry. In some places it is either sure that there is another consumer for this entry (e.g. when it received this entry from active_entries_ map, it dooms it in STATE_DOOM_ENTRY) or if its the only consumer for this entry (e.g. for no-store requests). In many cases when its the only consumer it will also call DoneWritingToEntry after dooming which will ensure the entry is destroyed. This explanation helped me to correct the logic in DoneValidationWithEntry in the new patch when its not a match. I have added a condition to check that if this was the only active consumer then invoke DestroyEntryRestartPendingTransactions else invoke DoomEntryRestartPendingTransactions https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:700: if (!entry_alive || entry != entry_alive) On 2017/03/07 at 16:45:41, jkarlin wrote: > return entry_alive && entry_alive == entry; Done. Also renamed entry_alive to active_entry https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:846: bool new_transaction) { Refactored this function into 4 functions: - Always called from HCT for a new transaction AddTransactionToEntry() so we do not need a boolean for new transaction. This further calls AddTransactionToEntryImpl() which could also be called for queued transactions. This further calls either AddTransactionToEntryForRead() or AddTransactionToEntryForWrite(). AddTransactionToEntryForRead() is also called directly for validated_queue since they should only be processed for reading. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:868: // become the writer if there is no active writer at this time. On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > Is there a reason for it to take up the writer slot before validation is complete? I don't think we want to rule out another transaction coming in later but finishing validation first and starting to write. The writer slot works as a lock (in the existing code and in this CL). Presence of the writer decides whether a new reader can be added or not. Similarly, since we only allow one validating transaction at a time, validating_transaction acts as a lock that no other transaction can start validating. Thus the scenario that you mention : "another transaction coming in later but finishing validation first and starting to write." should never happen. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:890: ProcessPendingQueue(entry); On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > nit, suggestion: It feels like this routine might be clearer if (e.g.) assignment in pending queue in these last several lines was unconditional, and then in ProcessPendingQueue situations like the lack of writer and a non-empty pending_queue were detected, and ProcessPendingQueue pushed the state along. If you agree with this suggestion, you might want to scan the other functions to see if it applies; having a single or a small number of functions with state advancing logic can make that state advancing logic much easier to understand. > > (I suspect it's too much of a refactor, but if you want to and think it would be a good idea, I'd supporting converting all such routines into an AdvanceState() routine and really centralize the state logic.) I have refactored this function quite a bit in the next patch. The current AddTransactionToEntry() is now only called for new transactions and not queued transactions before I saw this comment. PTAL. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:905: entry->readers.insert(transaction); On 2017/03/07 at 16:45:40, jkarlin wrote: > We're adding it to readers, but I don't think the transactions mode is set to READ yet is it? Actually this should never happen. ConvertWriterToReader should have already done that and if not, it will be added to validated_queue here. Removing this case and adding comments to explain this in the new refactored code. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:929: entry->validated_queue.erase(i); On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > Can't it be in the validated queue and be the writer? In that situation, won't the writer remain set to the transaction? That should not be possible. validated_queue insertion happens in DoneValidationWithEntry only if writer is not the same as this. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:945: if (transaction == entry->writer || On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > Suggestion: I'd value a comment here, parallel to the ones above. Done. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:970: DCHECK_EQ(transaction, entry->writer); On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > Suggestion: Hoist this up tot he top of the routine and make it DCHECK(transaction == entry->writer || transaction == entry->validating_transacation). That way it acts as a contract check as well as documentation at the beginning of the routine as to the expected environment. Done. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:971: entry->writer = nullptr; On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > Suggestion: If you take the above suggestion, you could hoist the "if (transaction == entry->writer) entry->writer = nullptr" up to the beginning of the routine and eliminate this line. done https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.cc#... net/http/http_cache.cc:1208: void HttpCache::DoomEntryRestartPendingTransactions(ActiveEntry* entry) { On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > This function and the next one look simple and are only called from one place. What's your thinking about having them as separate functions rather than hoisting them into their callers? (This may be my focussing on API cleanliness at the expense of other forms, but if so I'd like to understand the other advantages I'm missing.) In the new patch DestroyEntryRestartPendingTransactions() now has 2 callers, so that's fine. My thinking on having separate functions was to make it easier to add more callers for them in future which might or might not happen for subsequent shared writing CLs. So basically , yes, API cleanliness was the major advantage here. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:241: class WorkItem; On 2017/03/07 at 17:22:59, jkarlin wrote: > On 2017/03/07 16:45:41, jkarlin wrote: > > This isn't alphabetical. > > I actually don't know that they need to be alphabetical, but it's nicer if they are :) Done. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:251: class ActiveEntry { On 2017/03/07 at 16:45:42, jkarlin wrote: > If it's a class the members need to be private and have a _ suffix. This is still reasonable as a struct. Reverted it to being a struct. I initially thought it would have a weak_ptr_ and thus perhaps better as a class, but I didn't need it later. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:272: // Transactions waiting to be added to entry. On 2017/03/08 at 19:11:23, Randy Smith (Not in Mondays) wrote: > On 2017/03/07 22:54:30, Randy Smith (Not in Mondays) wrote: > > nit: Thank you for the extra comments; they help! Would you be willing to > > expand them with the queue/element transition pathways? I'm guessing that > > transactions move from pending_queue -> validating_transaction -> > > validated_queue -> {writer/readers}, but I'm not sure and it would be wonderful > > to have that here. > > > > minor additional nit/suggestion: Ordering the members in the basic order a transaction goes through them might be useful. Done https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:273: TransactionList pending_queue; On 2017/03/07 at 16:45:41, jkarlin wrote: > pending_queue seems a bit vague now. Perhaps add_to_entry_queue? Changed. add_to_entry_queue seems more consistent with HCT's states STATE_ADD_TO_ENTRY and STATE_ADD_TO_ENTRY_COMPLETE which is when a transaction is in this queue. Also renaming will_process_pending_queue, ProcessPendingQueue and OnProcessPendingQueue since they also process validated_queue. Renaming them to will_process_waiting_transactions, ProcessWaitingTransactions and OnProcessWaitingTransactions respectively. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:334: // Returns true if given entry is still active. On 2017/03/07 at 16:45:41, jkarlin wrote: > What does it mean for an entry to be active? It's called "ActiveEntry" ;) > > Perhaps Doomed() or IsDoomed? Actually, it needs to search if the entry is alive, either as an active or doomed entry. Changed the implementation to reflect this and the name to IsEntryAlive() https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:371: // method returns ERR_CACHE_RACE to signal the transaction that it cannot be On 2017/03/07 at 16:45:41, jkarlin wrote: > I don't think this actually does return ERR_CACHE_RACE. Can you remove that part of the comment? done https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:373: // one (in this case, the entry is no longer valid). validation_done should On 2017/03/07 at 16:45:41, jkarlin wrote: > |validation_done|. We wrap variable names in ||'s. N/A now after refactoring. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:375: // phase, new_transaction should be passed as true if this transaction is a On 2017/03/07 at 16:45:41, jkarlin wrote: > same with |new_transaction| N/A now after refactoring. Removed these from the comment. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:379: bool validation_done = false, On 2017/03/07 at 16:45:41, jkarlin wrote: > nit: s/validation_done/validation_needed/ N/A now after refactoring. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:380: bool new_transaction = false); On 2017/03/07 at 16:45:41, jkarlin wrote: > nit: is_new_transaction to make it clear that it's a bool and not a transaction type N/A now after refactoring. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:380: bool new_transaction = false); On 2017/03/08 at 19:11:23, Randy Smith (Not in Mondays) wrote: > On 2017/03/07 16:45:41, jkarlin wrote: > > This method has few enough existing callers that I don't think we should use > > default values and instead update the callers. > > The style guide generally recommends against default arguments (which they equivalence to overloaded functions). See https://google.github.io/styleguide/cppguide.html#Function_Overloading and the following. Can you give a sense as to why this should be an exception? It's often better to have separate functions with descriptive names, since that makes it easier for people reading at the call site to understand exactly what's being done. Thanks for the link. Removed default arguments as part of refactoring this function. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:388: int OnValidationMatch(ActiveEntry* entry, On 2017/03/07 at 16:45:41, jkarlin wrote: > In keeping with the naming scheme, perhaps DoneWithValidation? You could combine these two functions into one. Done. Created DoneValidationWithEntry() and merged the match and no match cases. Also created a separate DoneValidationWriteEntry() for the case where this transaction is responsible for writing to the cache, thus eliminating the need for write_this_response. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:417: // transaction was added to readers. On 2017/03/07 at 22:54:30, Randy Smith (Not in Mondays) wrote: > It's not clear to me what this documentation means; it's sorta "Do this thing. If you don't do this thing, return false". Which makes it sound like false is returned in an error case. But if I look at the code, it's more that this function moves a transaction out of a state in which its either the writer or the validating transaction into whatever the right next state is (reader or pending), and we return true/false to indicate what state it's actually moved into. Could we change the name and documentation to make this clearer? Looking at the calling locations, I actually think "DoneValidatingEntry" may make sense as a name, with the doc being "Signals that a transaction's cache update (writing/validation) is done and it should be moved into its next state. Returns true if the entry is now a reader". > > This is a specific instance of a general issue that I care about a lot. I really want header comments to make sense from the viewpoint of the consumer *without* the consumer having to look at the code of the producing class. The producing class should implement some abstraction that's higher level than its member variables, and should talk in the comments on its methods in the language of that higher layer abstraction. If that isn't true, that's a layering violation. Sometimes that's the best thing (sometimes avoiding layering violations require way too much effort to make it worthwhile) but I'm usually going to treat comments that leak information about the class implementation the same way I'd treat code, i.e. with a "Is there a better way to do this?" question. > > As a secondary point, I do wonder why the HttpCache::Transaction can't switch to read mode without the transaction actually being in the readers set in the HttpCache::ActiveEntry. When we later finish up with a writer transaction, won't we check the modes of the transactions in pending_transactions before we set a new writer? So shouldn't it be safe to just unilaterally change the mode to READ in the transaction even if the cache didn't transition it into the readers queue? This would allow getting rid of the bool completely. -- We want to preserve their original mode till we are sure that they can be readers i.e. entry was successfully written to the cache. In case of a failure we want these transactions to start afresh and then we would need their original mode. HttpCache.SimpleGET_ManyWriters_CancelFirst is a test that crashes if we don't preserve their original modes. The implementation is now greatly simplified in the new patch and improved the documentation as well. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache.h#n... net/http/http_cache.h:458: // IO callbacks with ERR_CACHE_RACE. On 2017/03/07 at 16:45:41, jkarlin wrote: > Is this actually a cache race though? Seems like a misuse of the flag. This error code is used whenever we want the HCT state machine to restart. Its more like ERR_CACHE_ENTRY_CREATE_FAILURE https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:726: if (!reading_ && rv == OK) { On 2017/03/07 at 16:45:42, jkarlin wrote: > How many ways can we get here? Can we just call 'return WaitBeforeRead()' which either returns OK or sets the next state to STATE_WAIT_BEFORE_READ from the necessary locations? There are a lot of conditional paths that can lead to the end of Start() state machine, some of them being: all the places where SetUpEntryForRead, ConvertWriterToReader and PartialHeadersReceived is invoked. Instead of taking care of all of them and remembering to do it whenever a code change impacting that is done, its better to detach it from the "start" state machine and treat it as a separate "wait" state machine. I started out following the former approach but encountered a lot of failing tests while putting it here significantly simplified the design conceptually as well as in the implementation. It also means there does not need to be any careful ordering between calling this function and doing other things with the entry like doom/call DoneWithEntry, this always gets called after everything else required by the particular control flow is already done with the entry. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:732: read_buf_ = NULL; // Release the buffer before invoking the callback. On 2017/03/07 at 16:45:42, jkarlin wrote: > s/NULL/nullptr/ done https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1792: if (write_this_response_) { On 2017/03/07 at 16:45:42, jkarlin wrote: > Seems clearer to call entry_->writer == this && request_->method != "HEAD" here Apart from the conditions entry_->writer == this && request_->method != "HEAD", its also where we are in the state machine (state STATE_OVERWRITE_CACHED_RESPONSE) that matters. Removed write_this_response_ though as part of refactoring OnValidationMatch. Now I am calling a new function DoneValidationWriteEntry at the point where write_this_response_ was set to true and just returning from here if this transaction is the writer. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1793: cache_->OnValidationMatch(entry_, this, true); On 2017/03/07 at 16:45:42, jkarlin wrote: > OnValidationMatch(entry_, this, true /* write_this_response */); N/A after the code refactoring of OnValidationMatch. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1811: return cache_->OnValidationMatch(entry_, this, false); On 2017/03/07 at 16:45:42, jkarlin wrote: > OnValidationMatch(entry_, this, false /* write_this_response */); N/A after the code refactoring of OnValidationMatch. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1832: mode_ = READ; On 2017/03/07 at 16:45:42, jkarlin wrote: > I'd have thought that the existing code would have already marked this transaction as a reader if the validation matched. It looks like SetupEntryForRead does. But perhaps the 304 path does not? Should it? If so, then we can just DCHECK the mode here. Simplified DoWaitBeforeRead to not have an additional 304 case since ConvertWriterToReader() gets called for all cases where validation is a match. (I am still looking in the case when couldnt_conditionalize_request_ is set to true. I just want to make sure that if couldnt_conditionalize_request_ is set then we never have a validation match case) We cannot set the mode_ as READ when ConvertWriterToReader() gets called and could not add it as the reader because we need to retain the original mode until we are sure that the response data was written completely to the cache. If not , this transaction will be restarted and it will need to have its original mode then. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:1834: ResetNetworkTransaction(); On 2017/03/07 at 16:45:42, jkarlin wrote: > Likewise, shouldn't this be taken care of already? I am looking into this. I am not sure if the couldnt_conditionalize_request_ case is handled. https://codereview.chromium.org/2721933002/diff/60001/net/http/http_cache_tra... net/http/http_cache_transaction.cc:2671: if (cache_->ConvertWriterToReader(entry_, this)) On 2017/03/07 at 16:45:42, jkarlin wrote: > I see, this is why we had to change mode_ = READ in DoWaitBeforeReadComplete. > > Perhaps we can set mode_ = READ reagardless? > > Also, ConvertWriterToReader now handles cases where the current transaction isn't actually a writer. Perhaps we should be calling a different function if it's not the writer? Simplified ConvertWriterToReader so it doesn't do anything if the current transaction isn't actually a writer because everything needed to be done will be done in DoWaitBeforeRead.
On 2017/03/08 at 19:11:23, rdsmith wrote: > Shivani: A high level design suggestion I want to run by you. It might not work (== moderately simplify the code) but if it does and you like it I'd rather do the review afterwards instead of before :-}. > > The pattern I find myself pulling towards for the HttpCache additions in this CL is something like: > * There is a state machine that advances an ActiveEntry > * There is a single function that executes this state machine. > * All the information needed to figure out how to advance the ActiveEntry is in the ActiveEntry or in the transactions in the ActiveEntry. > > This would mean that OnProcessPendingQueue & OnProcessParallelValidation could be combined, and I believe that state transitions in other parts of the code could be simplified. (e.g. if a transaction was done being validated you'd put it on the validated queue and invoke the state advancement machinery--no testing of writer or readers to figure out what happens next.) And it would mean evaluating the state machine would be much simpler, because there would just be one section of the code to read. > > I think this might help with AddTransactionToEntry and its arguments as well; specifically, this suggestion would mean that validated_queue would be (in some fashion) embodied on the transaction and hence wouldn't need to be an argument. > > If you see problems with this suggestion, let me know, since they'll almost certainly be design issues I'll need to take into account in the review anyway :-}. > > I'm going to continue on and review the cache transaction, but I'll hold off on further review of HttpCache until I hear back from you on this. > Josh mentioned the same feedback yesterday! I don't see any design concerns to this except, I think there are pros and cons of having a state machine: pro: Most functions will just resort to calling the state machine functions to assign the transaction and advance/regress the state. So it will be a single point where everything will be. I think this is a strong advantage. Just for completeness, there will need to be an AdvanceState() and also a corresponding RegressState() when a transaction needs to be removed. It will also help in letting the transactions not be overloaded to act as states as it is currently being done. con: State would be an additional variable to keep track of whenever a transaction is added/removed/changes its mode. I am hoping this should not lead to maintainability issues and I am writing a patch to verify this.
On 2017/03/09 15:28:17, shivanisha wrote: > On 2017/03/08 at 19:11:23, rdsmith wrote: > > Shivani: A high level design suggestion I want to run by you. It might not > work (== moderately simplify the code) but if it does and you like it I'd rather > do the review afterwards instead of before :-}. > > > > The pattern I find myself pulling towards for the HttpCache additions in this > CL is something like: > > * There is a state machine that advances an ActiveEntry > > * There is a single function that executes this state machine. > > * All the information needed to figure out how to advance the ActiveEntry is > in the ActiveEntry or in the transactions in the ActiveEntry. > > > > This would mean that OnProcessPendingQueue & OnProcessParallelValidation could > be combined, and I believe that state transitions in other parts of the code > could be simplified. (e.g. if a transaction was done being validated you'd put > it on the validated queue and invoke the state advancement machinery--no testing > of writer or readers to figure out what happens next.) And it would mean > evaluating the state machine would be much simpler, because there would just be > one section of the code to read. > > > > I think this might help with AddTransactionToEntry and its arguments as well; > specifically, this suggestion would mean that validated_queue would be (in some > fashion) embodied on the transaction and hence wouldn't need to be an argument. > > > > > If you see problems with this suggestion, let me know, since they'll almost > certainly be design issues I'll need to take into account in the review anyway > :-}. > > > > I'm going to continue on and review the cache transaction, but I'll hold off > on further review of HttpCache until I hear back from you on this. > > > > Josh mentioned the same feedback yesterday! > > I don't see any design concerns to this except, I think there are pros and cons > of having a state machine: > pro: Most functions will just resort to calling the state machine functions to > assign the transaction and advance/regress the state. So it will be a single > point where everything will be. I think this is a strong advantage. > Just for completeness, there will need to be an AdvanceState() and also a > corresponding RegressState() when a transaction needs to be removed. > It will also help in letting the transactions not be overloaded to act as states > as it is currently being done. > > con: State would be an additional variable to keep track of whenever a > transaction is added/removed/changes its mode. I am hoping this should not lead > to maintainability issues and I am writing a patch to verify this. To what extent is the state of a transaction (in the context of the ActiveEntry) not implicit in the information already stored in the transaction?
On 2017/03/09 at 16:26:22, rdsmith wrote: > On 2017/03/09 15:28:17, shivanisha wrote: > > On 2017/03/08 at 19:11:23, rdsmith wrote: > > > Shivani: A high level design suggestion I want to run by you. It might not > > work (== moderately simplify the code) but if it does and you like it I'd rather > > do the review afterwards instead of before :-}. > > > > > > The pattern I find myself pulling towards for the HttpCache additions in this > > CL is something like: > > > * There is a state machine that advances an ActiveEntry > > > * There is a single function that executes this state machine. > > > * All the information needed to figure out how to advance the ActiveEntry is > > in the ActiveEntry or in the transactions in the ActiveEntry. > > > > > > This would mean that OnProcessPendingQueue & OnProcessParallelValidation could > > be combined, and I believe that state transitions in other parts of the code > > could be simplified. (e.g. if a transaction was done being validated you'd put > > it on the validated queue and invoke the state advancement machinery--no testing > > of writer or readers to figure out what happens next.) And it would mean > > evaluating the state machine would be much simpler, because there would just be > > one section of the code to read. > > > > > > I think this might help with AddTransactionToEntry and its arguments as well; > > specifically, this suggestion would mean that validated_queue would be (in some > > fashion) embodied on the transaction and hence wouldn't need to be an argument. > > > > > > > > If you see problems with this suggestion, let me know, since they'll almost > > certainly be design issues I'll need to take into account in the review anyway > > :-}. > > > > > > I'm going to continue on and review the cache transaction, but I'll hold off > > on further review of HttpCache until I hear back from you on this. > > > > > > > Josh mentioned the same feedback yesterday! > > > > I don't see any design concerns to this except, I think there are pros and cons > > of having a state machine: > > pro: Most functions will just resort to calling the state machine functions to > > assign the transaction and advance/regress the state. So it will be a single > > point where everything will be. I think this is a strong advantage. > > Just for completeness, there will need to be an AdvanceState() and also a > > corresponding RegressState() when a transaction needs to be removed. > > It will also help in letting the transactions not be overloaded to act as states > > as it is currently being done. > > > > con: State would be an additional variable to keep track of whenever a > > transaction is added/removed/changes its mode. I am hoping this should not lead > > to maintainability issues and I am writing a patch to verify this. > > To what extent is the state of a transaction (in the context of the ActiveEntry) not implicit in the information already stored in the transaction? A transaction just knows its own mode and state which can mostly give it the knowledge of active entry's state except when active entry has things going on in parallel. States of the ActiveEntry may be one the following: NONE, VALIDATING, VALIDATING_WRITE, (validation and writing going on in parallel) WRITE, READ, VALIDATING_READ (validation and read going on in parallel)
On 2017/03/09 16:33:00, shivanisha wrote: > On 2017/03/09 at 16:26:22, rdsmith wrote: > > On 2017/03/09 15:28:17, shivanisha wrote: > > > On 2017/03/08 at 19:11:23, rdsmith wrote: > > > > Shivani: A high level design suggestion I want to run by you. It might > not > > > work (== moderately simplify the code) but if it does and you like it I'd > rather > > > do the review afterwards instead of before :-}. > > > > > > > > The pattern I find myself pulling towards for the HttpCache additions in > this > > > CL is something like: > > > > * There is a state machine that advances an ActiveEntry > > > > * There is a single function that executes this state machine. > > > > * All the information needed to figure out how to advance the ActiveEntry > is > > > in the ActiveEntry or in the transactions in the ActiveEntry. > > > > > > > > This would mean that OnProcessPendingQueue & OnProcessParallelValidation > could > > > be combined, and I believe that state transitions in other parts of the code > > > could be simplified. (e.g. if a transaction was done being validated you'd > put > > > it on the validated queue and invoke the state advancement machinery--no > testing > > > of writer or readers to figure out what happens next.) And it would mean > > > evaluating the state machine would be much simpler, because there would just > be > > > one section of the code to read. > > > > > > > > I think this might help with AddTransactionToEntry and its arguments as > well; > > > specifically, this suggestion would mean that validated_queue would be (in > some > > > fashion) embodied on the transaction and hence wouldn't need to be an > argument. > > > > > > > > > > > If you see problems with this suggestion, let me know, since they'll > almost > > > certainly be design issues I'll need to take into account in the review > anyway > > > :-}. > > > > > > > > I'm going to continue on and review the cache transaction, but I'll hold > off > > > on further review of HttpCache until I hear back from you on this. > > > > > > > > > > Josh mentioned the same feedback yesterday! > > > > > > I don't see any design concerns to this except, I think there are pros and > cons > > > of having a state machine: > > > pro: Most functions will just resort to calling the state machine functions > to > > > assign the transaction and advance/regress the state. So it will be a single > > > point where everything will be. I think this is a strong advantage. > > > Just for completeness, there will need to be an AdvanceState() and also a > > > corresponding RegressState() when a transaction needs to be removed. > > > It will also help in letting the transactions not be overloaded to act as > states > > > as it is currently being done. > > > > > > con: State would be an additional variable to keep track of whenever a > > > transaction is added/removed/changes its mode. I am hoping this should not > lead > > > to maintainability issues and I am writing a patch to verify this. > > > > To what extent is the state of a transaction (in the context of the > ActiveEntry) not implicit in the information already stored in the transaction? > > A transaction just knows its own mode and state which can mostly give it the > knowledge of active entry's state except when active entry has things going on > in parallel. States of the ActiveEntry may be one the following: > NONE, > VALIDATING, > VALIDATING_WRITE, (validation and writing going on in parallel) > WRITE, > READ, > VALIDATING_READ (validation and read going on in parallel) There's two different meanings of "state" going on here. What I mean by "The state of the transaction in the context of the active entry" is "What queues it's on or member variables it's in". The state of the active entry itself is one of the values you cite above. But the state of the active entry is implicit in what it's member variables are (i.e. if there's a validating transaction and not writer, it's validating). We don't need a new variable, or at least, from a high level conceptual place I don't think we should? The state of a transaction in the context of the active entry is encompassed in what queues or member variables it's assigned to. What I was gesturing at is that there can be times (AddTransactionToEntry) when we're attempting to assign a transaction to the correct queue/member variable. In that situation, is there enough information in the transaction to figure out where in the entry it should go? If not, what information is missing?
On 2017/03/09 at 16:52:16, rdsmith wrote: > On 2017/03/09 16:33:00, shivanisha wrote: > > On 2017/03/09 at 16:26:22, rdsmith wrote: > > > On 2017/03/09 15:28:17, shivanisha wrote: > > > > On 2017/03/08 at 19:11:23, rdsmith wrote: > > > > > Shivani: A high level design suggestion I want to run by you. It might > > not > > > > work (== moderately simplify the code) but if it does and you like it I'd > > rather > > > > do the review afterwards instead of before :-}. > > > > > > > > > > The pattern I find myself pulling towards for the HttpCache additions in > > this > > > > CL is something like: > > > > > * There is a state machine that advances an ActiveEntry > > > > > * There is a single function that executes this state machine. > > > > > * All the information needed to figure out how to advance the ActiveEntry > > is > > > > in the ActiveEntry or in the transactions in the ActiveEntry. > > > > > > > > > > This would mean that OnProcessPendingQueue & OnProcessParallelValidation > > could > > > > be combined, and I believe that state transitions in other parts of the code > > > > could be simplified. (e.g. if a transaction was done being validated you'd > > put > > > > it on the validated queue and invoke the state advancement machinery--no > > testing > > > > of writer or readers to figure out what happens next.) And it would mean > > > > evaluating the state machine would be much simpler, because there would just > > be > > > > one section of the code to read. > > > > > > > > > > I think this might help with AddTransactionToEntry and its arguments as > > well; > > > > specifically, this suggestion would mean that validated_queue would be (in > > some > > > > fashion) embodied on the transaction and hence wouldn't need to be an > > argument. > > > > > > > > > > > > > > If you see problems with this suggestion, let me know, since they'll > > almost > > > > certainly be design issues I'll need to take into account in the review > > anyway > > > > :-}. > > > > > > > > > > I'm going to continue on and review the cache transaction, but I'll hold > > off > > > > on further review of HttpCache until I hear back from you on this. > > > > > > > > > > > > > Josh mentioned the same feedback yesterday! > > > > > > > > I don't see any design concerns to this except, I think there are pros and > > cons > > > > of having a state machine: > > > > pro: Most functions will just resort to calling the state machine functions > > to > > > > assign the transaction and advance/regress the state. So it will be a single > > > > point where everything will be. I think this is a strong advantage. > > > > Just for completeness, there will need to be an AdvanceState() and also a > > > > corresponding RegressState() when a transaction needs to be removed. > > > > It will also help in letting the transactions not be overloaded to act as > > states > > > > as it is currently being done. > > > > > > > > con: State would be an additional variable to keep track of whenever a > > > > transaction is added/removed/changes its mode. I am hoping this should not > > lead > > > > to maintainability issues and I am writing a patch to verify this. > > > > > > To what extent is the state of a transaction (in the context of the > > ActiveEntry) not implicit in the information already stored in the transaction? > > > > A transaction just knows its own mode and state which can mostly give it the > > knowledge of active entry's state except when active entry has things going on > > in parallel. States of the ActiveEntry may be one the following: > > NONE, > > VALIDATING, > > VALIDATING_WRITE, (validation and writing going on in parallel) > > WRITE, > > READ, > > VALIDATING_READ (validation and read going on in parallel) > > There's two different meanings of "state" going on here. What I mean by "The state of the transaction in the context of the active entry" is "What queues it's on or member variables it's in". The state of the active entry itself is one of the values you cite above. But the state of the active entry is implicit in what it's member variables are (i.e. if there's a validating transaction and not writer, it's validating). We don't need a new variable, or at least, from a high level conceptual place I don't think we should? > > The state of a transaction in the context of the active entry is encompassed in what queues or member variables it's assigned to. What I was gesturing at is that there can be times (AddTransactionToEntry) when we're attempting to assign a transaction to the correct queue/member variable. In that situation, is there enough information in the transaction to figure out where in the entry it should go? If not, what information is missing? The information that a transaction itself misses is another transaction's place in the ActuveEntry. So e.g. if a transaction is done validating and wants to become a reader, it does not know if it can, based on whether another transaction is still the writer and thus writing to the entry.
On 2017/03/09 17:00:03, shivanisha wrote: > On 2017/03/09 at 16:52:16, rdsmith wrote: > > On 2017/03/09 16:33:00, shivanisha wrote: > > > On 2017/03/09 at 16:26:22, rdsmith wrote: > > > > On 2017/03/09 15:28:17, shivanisha wrote: > > > > > On 2017/03/08 at 19:11:23, rdsmith wrote: > > > > > > Shivani: A high level design suggestion I want to run by you. It > might > > > not > > > > > work (== moderately simplify the code) but if it does and you like it > I'd > > > rather > > > > > do the review afterwards instead of before :-}. > > > > > > > > > > > > The pattern I find myself pulling towards for the HttpCache additions > in > > > this > > > > > CL is something like: > > > > > > * There is a state machine that advances an ActiveEntry > > > > > > * There is a single function that executes this state machine. > > > > > > * All the information needed to figure out how to advance the > ActiveEntry > > > is > > > > > in the ActiveEntry or in the transactions in the ActiveEntry. > > > > > > > > > > > > This would mean that OnProcessPendingQueue & > OnProcessParallelValidation > > > could > > > > > be combined, and I believe that state transitions in other parts of the > code > > > > > could be simplified. (e.g. if a transaction was done being validated > you'd > > > put > > > > > it on the validated queue and invoke the state advancement machinery--no > > > testing > > > > > of writer or readers to figure out what happens next.) And it would > mean > > > > > evaluating the state machine would be much simpler, because there would > just > > > be > > > > > one section of the code to read. > > > > > > > > > > > > I think this might help with AddTransactionToEntry and its arguments > as > > > well; > > > > > specifically, this suggestion would mean that validated_queue would be > (in > > > some > > > > > fashion) embodied on the transaction and hence wouldn't need to be an > > > argument. > > > > > > > > > > > > > > > > > If you see problems with this suggestion, let me know, since they'll > > > almost > > > > > certainly be design issues I'll need to take into account in the review > > > anyway > > > > > :-}. > > > > > > > > > > > > I'm going to continue on and review the cache transaction, but I'll > hold > > > off > > > > > on further review of HttpCache until I hear back from you on this. > > > > > > > > > > > > > > > > Josh mentioned the same feedback yesterday! > > > > > > > > > > I don't see any design concerns to this except, I think there are pros > and > > > cons > > > > > of having a state machine: > > > > > pro: Most functions will just resort to calling the state machine > functions > > > to > > > > > assign the transaction and advance/regress the state. So it will be a > single > > > > > point where everything will be. I think this is a strong advantage. > > > > > Just for completeness, there will need to be an AdvanceState() and also > a > > > > > corresponding RegressState() when a transaction needs to be removed. > > > > > It will also help in letting the transactions not be overloaded to act > as > > > states > > > > > as it is currently being done. > > > > > > > > > > con: State would be an additional variable to keep track of whenever a > > > > > transaction is added/removed/changes its mode. I am hoping this should > not > > > lead > > > > > to maintainability issues and I am writing a patch to verify this. > > > > > > > > To what extent is the state of a transaction (in the context of the > > > ActiveEntry) not implicit in the information already stored in the > transaction? > > > > > > A transaction just knows its own mode and state which can mostly give it the > > > knowledge of active entry's state except when active entry has things going > on > > > in parallel. States of the ActiveEntry may be one the following: > > > NONE, > > > VALIDATING, > > > VALIDATING_WRITE, (validation and writing going on in parallel) > > > WRITE, > > > READ, > > > VALIDATING_READ (validation and read going on in parallel) > > > > There's two different meanings of "state" going on here. What I mean by "The > state of the transaction in the context of the active entry" is "What queues > it's on or member variables it's in". The state of the active entry itself is > one of the values you cite above. But the state of the active entry is implicit > in what it's member variables are (i.e. if there's a validating transaction and > not writer, it's validating). We don't need a new variable, or at least, from a > high level conceptual place I don't think we should? > > > > The state of a transaction in the context of the active entry is encompassed > in what queues or member variables it's assigned to. What I was gesturing at is > that there can be times (AddTransactionToEntry) when we're attempting to assign > a transaction to the correct queue/member variable. In that situation, is there > enough information in the transaction to figure out where in the entry it should > go? If not, what information is missing? > > The information that a transaction itself misses is another transaction's place > in the ActuveEntry. So e.g. if a transaction is done validating and wants to > become a reader, it does not know if it can, based on whether another > transaction is still the writer and thus writing to the entry. I think we're talking past each other--maybe we should grab a room tomorrow? Your "con" was that an extra variable to track state might be used. My response was intended to be "What state isn't already implied by the information in the combination of HttpCache::Transaction and ActiveEntry?" (which is the information that's generally available when you need to make state decisions) . It sounds like the information you mention above is present in ActiveEntry.
A high level comment and some cleanup suggestions to http_cache.h https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:282: // the writer) How about the following design for the progression of ActiveEntry states? How far off is this from what is already implemented? 1. Writing Response Headers 2. Writing Response Body 3. Reading The first two states are optional (e.g., if no writing is necessary). But the state can only move forward (e.g., if the writing fails in state one, abort entry and restart transactions, else abort entry and all transactions). Once in the Reading state, no writing should occur (including writing back 304 response headers of validating transaction?), and any transaction that doesn't validate leaves and moves on to a new transaction. While in states 2 or 3, parallel validation may occur. https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:401: Transaction* transaction); The above 3 methods don't need to be class methods. They can live in the cc file in the anonymous namespace. https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:420: void DoneValidationWriteEntry(ActiveEntry* entry, Transaction* transaction); So at this point the Writer is done with the response (e.g., it has written or validated the response headers)? In which case, let's call this one something like DoneWritingResponseHeadersToEntry. https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:435: void DoneWritingToEntryProcessOtherTransactions(ActiveEntry* entry, Should be in anonymous namespace? https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:466: void ProcessWaitingTransactions(ActiveEntry* entry); Should be in anonymous namespace? https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:470: void ProcessParallelValidation(ActiveEntry* entry); Should be in anonymous namespace? https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:474: void OnProcessWaitingTransactions(const std::string& key, ActiveEntry* entry); Should be in anonymous namespace? https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:476: void OnProcessParallelValidation(const std::string& key, ActiveEntry* entry); Should be in anonymous namespace? https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:485: TransactionList GetAllPendingTransactions(ActiveEntry* entry); Should be in anonymous namespace? Also, why is this in the "Events" section of the header? https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:489: void RestartPendingTransactions(const TransactionList& list); Should be in anonymous namespace?
https://codereview.chromium.org/2721933002/diff/100001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/100001/net/http/http_cache.h#... net/http/http_cache.h:279: }; Suggest: NONE, WRITING_RESPONSE_HEADERS, WRITING_RESPONSE_BODY, READING VALIDATING doesn't make sense to me. What if it's a new cache entry and the writer is just writing out the response headers, there is no validation involved. Is it necessary to differentiate the validation states? The key moments are determining when it's safe to validate in parallel and when readers can start.
Patch 3 is very rough: not compiling, but I just wanted to make sure if the extra code for making a state machine in ActiveEntry is worth it. Had a chat discussion with Josh and his suggestion is that it does make things clearer. Going ahead with it.
This patch has the Active Entry state machine. Testing it is in progress but uploading it to be used for discussions.
This patch follows the f2f discussion that we had and has the following major changes: - Removes ConvertWriterToReader so that all scenarios of completing the headers phase are handled in one single function. That function is now called DoneResponseHeaders. Renamed it from DoneValidationWithEntry because for the 1st response, its not really validation but getting the headers for the first time. - Also renamed other things like validated_queue which is now done_headers_queue because of the same reason. - Makes the state transitions simpler and consistent for all transactions. Here is the simplified state transition: Write mode transactions: add_to_entry_queue-> headers_transaction -> done_headers_queue -> writer add_to_entry_queue-> headers_transaction -> done_headers_queue -> readers (once the data is written to the cache by another writer) Read only transactions: add_to_entry_queue-> readers (once the data is written to the cache by the writer) So being in add_to_entry_queue or done_headers_queue is no longer optional. - Any event has 2 parts in the entry: impact on the transaction that caused the event and impact on other transactions. The first part is in the calling location like DoneResponseHeaders, DoneWritingToEntry etc. and the 2nd part is in either ProcessQueuedTransactions (for success) or ProcessEntryFailure (for failure). ProcessQueuedTransactions like earlier posts a task to avoid reentrancy and ProcessEntryFailure(as earlier) is synchronous as it leads to either dooming/destroying the entry. - AddTransactionToEntry is now only invoked for adding a new transaction and for processing a queued transaction either ProcessAddToEntryTransaction or ProcessDoneHeadersTransaction is invoked based on which queue needs to be processed. Let me know if there's any clarification needed either through chat or we can do a quick VC. Thanks! https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:401: Transaction* transaction); On 2017/03/09 at 17:42:57, jkarlin wrote: > The above 3 methods don't need to be class methods. They can live in the cc file in the anonymous namespace. N/A after recent refactoring. https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:420: void DoneValidationWriteEntry(ActiveEntry* entry, Transaction* transaction); On 2017/03/09 at 17:42:56, jkarlin wrote: > So at this point the Writer is done with the response (e.g., it has written or validated the response headers)? In which case, let's call this one something like DoneWritingResponseHeadersToEntry. Renamed to DoneResponseHeaders because in case of no match, the headers are not written to the entry. https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:435: void DoneWritingToEntryProcessOtherTransactions(ActiveEntry* entry, On 2017/03/09 at 17:42:56, jkarlin wrote: > Should be in anonymous namespace? This function no longer exists https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:485: TransactionList GetAllPendingTransactions(ActiveEntry* entry); On 2017/03/09 at 17:42:57, jkarlin wrote: > Should be in anonymous namespace? Also, why is this in the "Events" section of the header? Since ActiveEntry is a private structure in HttpCache, cannot keep it in anonymous namespace. https://codereview.chromium.org/2721933002/diff/80001/net/http/http_cache.h#n... net/http/http_cache.h:489: void RestartPendingTransactions(const TransactionList& list); On 2017/03/09 at 17:42:57, jkarlin wrote: > Should be in anonymous namespace? done
In addition to above, this patch also implements failing a headers_transaction if the writer fails. Since the headers_transaction may be in a waiting state, a flag will be set and on its IO callback invocation the transaction will be restarted based on the flag being true. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:870: // A transaction either starts validation or waits. "A transaction either starts headers phase or waits." https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:934: HttpCache::TransactionList HttpCache::GetAllPendingTransactions( will rename this to GetAllQueuedTransactions and send TransactionList& as an output argument instead of return value to avoid any copies. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:1020: // Transaction is removed in the validation phase. "Transaction is removed in the headers phase." https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:407: // either on validating existing headers/ skippig validation or getting new *skipping https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:443: // void ConvertWriterToReader(ActiveEntry* entry, Transaction* transaction); Will remove this commented code.
https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:265: bool IsReader(Transaction* transaction); Will remove this function since it is no longer invoked. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:409: // it was new headers. *or headers were retrieved the first time. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:432: bool success); Will remove this function declaration as it is no longer needed. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1832: void HttpCache::Transaction::RestartAfterValidationStarted() { DCHECK(!reading_);
I like this shift--thanks for implementing it! First round of comments. Two high level requests: * It looks like you're going to be removing some functions; could you give a heads up when you do so? On of the things I'm trying to evaluate is what the API surface change looks like, and reviewing that after the removal will help. * When we're next both in the office, I'd like to talk with you (and Josh if he's interested) around the extra code in DoLoop/DoLoopImpl in HttpCache::Transaction. It's moving away from the standard DoLoop() pattern, and I'd like to explore if there are other ways to manage that. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:57: void RestartPendingTransactions(const net::HttpCache::TransactionList& list) { It looks to me as if this function is the reason why TransactionList and TransactionSet are now public on HttpCache. Is that correct? If so, it doesn't strike me as worth it--I think more restricted access would be valuable enough to make it worth either making this function a static member of HttpCache or hoisting it's usage into the calling locations (it isn't particularly long). WDYT? I'm not invested in the change, but do prefer private to public for all the usual reasons so I'd like to understand the counter argument before accepting it. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:60: transaction->io_callback().Run(net::ERR_CACHE_RACE); To me, calling a callback on a transaction is saying to a consumer "I've completed the work you requested on this transaction, and this is the result". Given that it's effectively handing responsibility for the transactions back to the consumer, I'd like it to be explicitly coupled with removing any references to the transaction at this level. Would you be willing to rewrite this routine so it modifies the lists passed to it s.t. the transactions for which the io_callback() is being called are removed from the passed in list? https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:864: // possible that the headers_transaction dooms the entry if its not a match. Suggestion: Based on the state transitions that you document in the header file, this conditional is obvious--the next step for transactions in the add_to_entry_queue is to headers_transaction, and if that slot is filled, the transition can't occur. So I'd think this comment belongs in documenting that state transition rather than here? https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:872: entry->headers_transaction = transaction; Suggestion: Hoist this conditional up above the if(entry->writer) conditional above? By the state transitions sketched out in the header, the next step for a WRITE mode transaction is the headers_transaction, so there isn't a conceptual need to distinguish between whether there's a writer or not? That's only needed if there's a reader. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:889: ProcessQueuedTransactions(entry); Suggestion: This routine is already executed after a PostTask; is another one needed? I.e. can this be handled with a loop in the calling routine? https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:914: transaction->io_callback().Run(OK); Help me understand why this callback is being run? I presume we want to run these callbacks when some consumer visible change occurs on the transaction. But the consumer visible change in this particular case is that we're done with the headers (right?), so shouldn't this be in the code that triggers the queue processing (because it has to do with the effect on the single transaction, not on the other transactions, and because it's associated with the transition from headers_transaction -> done_headers_queue, not from done_headers_queue onwards?) https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:998: DCHECK(!entry->headers_transaction); Why this invariant? What if we have the writer and the headers_transaction completing quickly, one after the other, and the PostTasks merging so that when we reach this point both are empty? https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:1004: ProcessAddToEntryTransaction(entry); Comment (not even a suggestion): If I were writing this code, I would inline ProcessAddToEntryTransaction and ProcessDoneHeadersTransaction, basically because I think the virtue of seeing all the queue manipulation code in the same place outweighs the virtue of having separate functions. The fact that when reviewing these functions I found myself repeatedly jumping back and forth between OnProcessQueuedTransactions and its helpers also biases me this way. But I think this is basically an aesthetic issue, so if you've thought about the two options and decided that you preferred this, I'm happy with that. Request: However, I would like to have the functions near each other in the .cc file. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:404: void ProcessDoneHeadersTransaction(ActiveEntry* entry); Suggestion: Include in the comments that the above two functions are both helper functions for OnProcessQueuedTransactions? They aren't called anywhere else, so I think it's both fair and useful for the reader to frame them as such. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1802: if (entry_->writer == this || request_->method == "HEAD") I'm a bit concerned about races here. The transition to writer from validating_transaction completes after a PostTask. Can we be in a situation where that happens after this test, and if so, what's the result? https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1805: // If |this| is in readers, then no need to wait. This comment doesn't seem to fit the following lines? https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1806: if (entry_->headers_transaction == this) When does this happen? Naively, I think of WAIT_BEFORE_READ as being the state reflection in HttpCache::Transaction between the completion of being a headers_transaction and the start of being a reader/writer, so we should always be in the done_headers_queue at this point. What am I missing?
This patch addresses Randy's latest feedback and some self review feedback. The latest code removes the following functions from earlier patches: - ActiveEntry::IsReader as this is no longer needed after the removal of HttpCache::ConvertWriterToReader - HttpCache::DoneWritingToEntryProcessOtherTransactions as the logic in this is now part of ProcessEntryFailure https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:57: void RestartPendingTransactions(const net::HttpCache::TransactionList& list) { On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > It looks to me as if this function is the reason why TransactionList and TransactionSet are now public on HttpCache. Is that correct? If so, it doesn't strike me as worth it--I think more restricted access would be valuable enough to make it worth either making this function a static member of HttpCache or hoisting it's usage into the calling locations (it isn't particularly long). WDYT? > > I'm not invested in the change, but do prefer private to public for all the usual reasons so I'd like to understand the counter argument before accepting it. Good catch, since this change not only made TransactionList and TransactionSet public but also the Transaction class itself. I agree, its not worth it. Making the logic here inline in the calling location and reverting the private scopes of Transaction, TransactionList and TransactionSet. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:60: transaction->io_callback().Run(net::ERR_CACHE_RACE); On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > To me, calling a callback on a transaction is saying to a consumer "I've completed the work you requested on this transaction, and this is the result". Given that it's effectively handing responsibility for the transactions back to the consumer, I'd like it to be explicitly coupled with removing any references to the transaction at this level. Would you be willing to rewrite this routine so it modifies the lists passed to it s.t. the transactions for which the io_callback() is being called are removed from the passed in list? This routine is now removed and callback is invoked in the calling location. Renamed GetAllPendingTransactions as RemoveAllQueuedTransactions so that it is clear that the function is also clearing the queues and also clarified in its documentation. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:864: // possible that the headers_transaction dooms the entry if its not a match. On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > Suggestion: Based on the state transitions that you document in the header file, this conditional is obvious--the next step for transactions in the add_to_entry_queue is to headers_transaction, and if that slot is filled, the transition can't occur. So I'd think this comment belongs in documenting that state transition rather than here? I agree that this comment is basically explaining "why" a transaction should wait when headers_transaction slot is filled. I wrote this here mainly to emphasize nuances of this situation like "the entry may be new or may already have a response body written to it" and answering questions like : why a read-only transaction can't start reading the already written response. Since these are implementation specific aspects, it seemed better for a reader to read while understanding the code in this function. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:870: // A transaction either starts validation or waits. On 2017/03/16 at 00:54:54, shivanisha wrote: > "A transaction either starts headers phase or waits." done https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:872: entry->headers_transaction = transaction; On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > Suggestion: Hoist this conditional up above the if(entry->writer) conditional above? By the state transitions sketched out in the header, the next step for a WRITE mode transaction is the headers_transaction, so there isn't a conceptual need to distinguish between whether there's a writer or not? That's only needed if there's a reader. Here I wanted to emphasize the conceptual differences between the 2 states when there is a writer or not, namely, either the entry is being written to (there is a writer) or during the start of an entry or when the response is already written (writer does not exist) https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:889: ProcessQueuedTransactions(entry); On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > Suggestion: This routine is already executed after a PostTask; is another one needed? I.e. can this be handled with a loop in the calling routine? Since additional queued transactions should only be processed in case of readers, adding it here makes the complete logic reside in the same place. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:914: transaction->io_callback().Run(OK); On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > Help me understand why this callback is being run? I presume we want to run these callbacks when some consumer visible change occurs on the transaction. But the consumer visible change in this particular case is that we're done with the headers (right?), so shouldn't this be in the code that triggers the queue processing (because it has to do with the effect on the single transaction, not on the other transactions, and because it's associated with the transition from headers_transaction -> done_headers_queue, not from done_headers_queue onwards?) This is indeed the callback of the "other" transaction which was earlier in done_headers_queue and now taken out of it. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:934: HttpCache::TransactionList HttpCache::GetAllPendingTransactions( On 2017/03/16 at 00:54:53, shivanisha wrote: > will rename this to GetAllQueuedTransactions and send TransactionList& as an output argument instead of return value to avoid any copies. Renamed it to RemoveAllQueuedTransactions to be clear. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:998: DCHECK(!entry->headers_transaction); On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > Why this invariant? What if we have the writer and the headers_transaction completing quickly, one after the other, and the PostTasks merging so that when we reach this point both are empty? Good catch, Thanks. Its possible that writer just completed writing to the entry and is thus null and there is a headers_transaction validating the headers. Removing this dcheck from here and also from ProcessDoneHeadersTransaction. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:1004: ProcessAddToEntryTransaction(entry); On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > Comment (not even a suggestion): If I were writing this code, I would inline ProcessAddToEntryTransaction and ProcessDoneHeadersTransaction, basically because I think the virtue of seeing all the queue manipulation code in the same place outweighs the virtue of having separate functions. The fact that when reviewing these functions I found myself repeatedly jumping back and forth between OnProcessQueuedTransactions and its helpers also biases me this way. But I think this is basically an aesthetic issue, so if you've thought about the two options and decided that you preferred this, I'm happy with that. > > Request: However, I would like to have the functions near each other in the .cc file. Moved the functions to be near each other. My main motivation to have separate functions is to let the reader have a high level understanding of the flow from reading OnProcessQueuedTransactions and then to dig deeper in the state transitions of a transaction by reading ProcessDoneHeadersTransaction and ProcessAddToEntryTransaction. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.cc... net/http/http_cache.cc:1020: // Transaction is removed in the validation phase. On 2017/03/16 at 00:54:53, shivanisha wrote: > "Transaction is removed in the headers phase." done https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:265: bool IsReader(Transaction* transaction); On 2017/03/16 at 15:43:26, shivanisha wrote: > Will remove this function since it is no longer invoked. done https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:404: void ProcessDoneHeadersTransaction(ActiveEntry* entry); On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > Suggestion: Include in the comments that the above two functions are both helper functions for OnProcessQueuedTransactions? They aren't called anywhere else, so I think it's both fair and useful for the reader to frame them as such. Done https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:407: // either on validating existing headers/ skippig validation or getting new On 2017/03/16 at 00:54:54, shivanisha wrote: > *skipping done https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:409: // it was new headers. On 2017/03/16 at 15:43:26, shivanisha wrote: > *or headers were retrieved the first time. done https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:432: bool success); On 2017/03/16 at 15:43:26, shivanisha wrote: > Will remove this function declaration as it is no longer needed. done https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache.h#... net/http/http_cache.h:443: // void ConvertWriterToReader(ActiveEntry* entry, Transaction* transaction); On 2017/03/16 at 00:54:54, shivanisha wrote: > Will remove this commented code. done
This patch addresses Randy's latest feedback in http_cache_transaction.cc This includes a new boolean variable is_wait_before_read_complete_ that is checked so that control does not enter STATE_WAIT_BEFORE_READ a second time. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1802: if (entry_->writer == this || request_->method == "HEAD") On 2017/03/17 at 18:13:06, Randy Smith (Not in Mondays) wrote: > I'm a bit concerned about races here. The transition to writer from validating_transaction completes after a PostTask. Can we be in a situation where that happens after this test, and if so, what's the result? Actually, after the state simplification since a transaction only becomes writer after this function, this condition can be removed. A related change: (We just have to make sure that after completing DoWaitBeforeReadComplete is executed, this function does not get executed again due to the DoLoop code. For that, I am adding a boolean is_wait_before_read_complete_ and checking it in DoLoop in addition to !reading_ before setting the state to STATE_WAIT_BEFORE_READ) https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1805: // If |this| is in readers, then no need to wait. On 2017/03/17 at 18:13:06, Randy Smith (Not in Mondays) wrote: > This comment doesn't seem to fit the following lines? Expanded the comment to be more clear. The if condition basically wants to exclude transactions that are already in readers, which can happen to transactions with read-only permissions. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1806: if (entry_->headers_transaction == this) On 2017/03/17 at 18:13:05, Randy Smith (Not in Mondays) wrote: > When does this happen? Naively, I think of WAIT_BEFORE_READ as being the state reflection in HttpCache::Transaction between the completion of being a headers_transaction and the start of being a reader/writer, so we should always be in the done_headers_queue at this point. What am I missing? At this point a transaction should always be a headers_transaction and will move to done_headers_queue in DoneResponseHeaders(). The only exception is the transactions that are read-only, they would be in readers at this point, thus this condition. https://codereview.chromium.org/2721933002/diff/140001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1832: void HttpCache::Transaction::RestartAfterValidationStarted() { On 2017/03/16 at 15:43:26, shivanisha wrote: > DCHECK(!reading_); done
This patch (DoLoop modified) has the following major changes over earlier patches: - DoLoop does not have the code for restarting of a headers_transaction. As discussed f2f, it is now moved to IO_Callback() - DoLoop does not have any special handling to move the state to STATE_WAIT_BEFORE_READ. As discussed f2f, it is now moved to the individual calling locations where state should transition to STATE_WAIT_BEFORE_READ. A dcheck is included in DoLoop() to check that Start() state machine should always exit in state STATE_WAIT_BEFORE_READ_COMPLETE to facilitate future development not to miss out this case. - Rebased with Josh's CL: https://codereview.chromium.org/2766953002/ - special cases handling added: If the response code is 401/407/416 the start machine is returning as OK to the consumer but in these cases we do not want the transaction to move on to done_headers_queue. It should still remain as headers_transaction till consumer invokes a RestartWithAuth API or destroys the transaction. - Destroying a headers_transaction which created the entry should also lead to destroying the disk entry and not just deactivating the active entry as this transaction was supposed to be writing the response. Added handling for that.
This patch uses TransitionToState after rebasing instead of setting next_state_ directly.
On 2017/03/23 at 15:03:57, shivanisha wrote: > This patch uses TransitionToState after rebasing instead of setting next_state_ directly. This patch combines rebasing with changes to my CL (using TransitionToState) as both seem related. Will keep the rebases as separate patches in future.
This patch fixes DoneWithEntry for headers_transaction so that cancel flag can propagate to DoneWritingToEntry as success or failure. For a headers_transaction which was supposed to write the response, cancel is set to true if its destroyed before it started writing to the response body. cancel flag basically implies "response is not completely written". https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1900: next_state_ = STATE_INIT_ENTRY; Did not use TransitionToState here as it can also be invoked directly from outside the DoLoop.
Some comments after looking at http_cache_transaction. https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:918: DCHECK(!entry_ || state == STATE_WAIT_BEFORE_READ_COMPLETE); DCHECK(reading_ || rv != OK || !entry_ || state == STATE_WAIT_BEFORE_READ_COMPLETE); https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:2770: // wait queue in DoWaitBeforeRead. Seems like this comment is unnecessary. https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... net/http/http_cache_transaction.h:166: void SetValidatingCannotProceed(); This isn't an HttpTransaction method (as the above are). Should have a newline above. Also, needs a comment since it's a public non-override method. https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:231: if (!cancel_request) Looks like cancel_request can't be true if !reading_ at this point, so how about: cancel_request |= !reading && created_entry_ && (...) and removing the conditional if (!cancel_request)? https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:232: cancel_request = !reading_ && created_entry_ && I find cancel_request hinging on whether we created an entry or not to be very strange. Is there any way we can be more consistent about cance_requests meaning instead of a specific subset of requests that canceled before finishing? https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1588: if (!reading_) prefer ternay here and the other cases like this: TransitionToState(reading_ ? STATE_NONE : STATE_WAIT_BEFORE_READ) https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1613: if (new_response->headers->response_code() != 304 && How do we know this transaction can't write to this entry? The conditional isn't obvious to me.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:918: DCHECK(!entry_ || state == STATE_WAIT_BEFORE_READ_COMPLETE); On 2017/03/23 at 18:37:38, jkarlin wrote: > DCHECK(reading_ || rv != OK || !entry_ || state == STATE_WAIT_BEFORE_READ_COMPLETE); Done . Also updated the comment. https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:2770: // wait queue in DoWaitBeforeRead. On 2017/03/23 at 18:37:38, jkarlin wrote: > Seems like this comment is unnecessary. Removed. Thanks. https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/220001/net/http/http_cache_tr... net/http/http_cache_transaction.h:166: void SetValidatingCannotProceed(); On 2017/03/23 at 18:37:38, jkarlin wrote: > This isn't an HttpTransaction method (as the above are). Should have a newline above. Also, needs a comment since it's a public non-override method. Done. https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:231: if (!cancel_request) On 2017/03/23 at 18:37:38, jkarlin wrote: > Looks like cancel_request can't be true if !reading_ at this point, so how about: > > cancel_request |= !reading && created_entry_ && (...) and removing the conditional if (!cancel_request)? N/A after the changes https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:232: cancel_request = !reading_ && created_entry_ && On 2017/03/23 at 18:37:38, jkarlin wrote: > I find cancel_request hinging on whether we created an entry or not to be very strange. Is there any way we can be more consistent about cance_requests meaning instead of a specific subset of requests that canceled before finishing? Removed cancel_request and changed the conditional. Either the transaction is a current writer or its an auth/416 case where it is not added to the done_headers_queue. https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1588: if (!reading_) On 2017/03/23 at 18:37:38, jkarlin wrote: > prefer ternay here and the other cases like this: > > TransitionToState(reading_ ? STATE_NONE : STATE_WAIT_BEFORE_READ) Done https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1613: if (new_response->headers->response_code() != 304 && On 2017/03/23 at 18:37:38, jkarlin wrote: > How do we know this transaction can't write to this entry? The conditional isn't obvious to me. It cannot write to the entry because either a writer or readers exist for the current entry and this validation is a no match case.
https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/240001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:232: cancel_request = !reading_ && created_entry_ && On 2017/03/29 at 03:39:30, shivanisha wrote: > On 2017/03/23 at 18:37:38, jkarlin wrote: > > I find cancel_request hinging on whether we created an entry or not to be very strange. Is there any way we can be more consistent about cance_requests meaning instead of a specific subset of requests that canceled before finishing? > > Removed cancel_request and changed the conditional. Either the transaction is a current writer or its an auth/416 case where it is not added to the done_headers_queue. *Removed created_request ...
https://codereview.chromium.org/2721933002/diff/280001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/280001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:220: new_response_->headers->response_code() == 416))); This still has special case logic. Thinking about how to simplify it further.
A function added in HttpCache to compute whether destroying a given transaction will lead to terminating the response body before it is complete.
HttpCacheTransaction is coming together. One last set of changes. The general idea is that the Transaction shouldn't be aware of its role in the ActiveEntry. It should just go on its merry way through its own state machine and let the ActiveEntry guide it (e.g., by delaying at certain points or changing its mode). https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1001: original_mode_ = mode_; There should be a newline above this as it doesn't have to do with the above comment. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1379: TransitionToState(reading_ ? STATE_NONE : STATE_WAIT_BEFORE_READ); Is this use of reading_ really a check to see if this transaction is actually the ActiveEntry's writer? If so, then let's just go to STATE_WAIT_BEFORE_READ and let the ActiveEntry synchronously tell the Transaction that it can read. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1832: DCHECK_NE(entry_->writer, this); The Transaction should be unaware of its role in the ActiveEntry. Nix this DCHECK. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1852: if (entry_->headers_transaction == this) The Transaction should be unaware of its role in the ActiveEntry. Just check if this Transaction is read-only. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1867: It looks like at this point all paths lead to return OK. One path leads to mode_ = READ and resetting the network transaction. Perhaps instead the HttpCache should force the Transaction into READ mode so that it doesn't have to infer its place in the ActiveEntry? https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:3199: if (!reading_ && validating_cannot_proceed_) { Same as above, let's not use reading_ to infer our state in the ActiveEntry. I think you can DCHECK(!reading_) however.
https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:992: bool HttpCache::IsCancelResponseBody(ActiveEntry* entry, The name IsCancelResponseBody doesn't make sense to me. Perhaps IsBackendEntryFinished? https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:995: bool cancel = (method != "HEAD"); The cancel variable is unnecessary and adds confusion. The following works just as well: if (method == "HEAD") return false; if (transaction == entry->writer) return true; ... https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:999: It seems like this function shouldn't be necessary. Can the Transaction really not tell: 1) If it has written bytes to the cache entry 2) If it isn't yet done writing bytes to the cache entry
Addressed Josh's feedback on HttpCache::Transaction. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1001: original_mode_ = mode_; On 2017/03/30 at 14:25:26, jkarlin wrote: > There should be a newline above this as it doesn't have to do with the above comment. Done. Also added a comment on why we need original_mode_ to be set here. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1379: TransitionToState(reading_ ? STATE_NONE : STATE_WAIT_BEFORE_READ); On 2017/03/30 at 14:25:26, jkarlin wrote: > Is this use of reading_ really a check to see if this transaction is actually the ActiveEntry's writer? If so, then let's just go to STATE_WAIT_BEFORE_READ and let the ActiveEntry synchronously tell the Transaction that it can read. Done. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1832: DCHECK_NE(entry_->writer, this); On 2017/03/30 at 14:25:26, jkarlin wrote: > The Transaction should be unaware of its role in the ActiveEntry. Nix this DCHECK. Removed , also because now we are allowing writers to be here and it will be treated as a no-op in DoneResponseHeaders. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1852: if (entry_->headers_transaction == this) On 2017/03/30 at 14:25:26, jkarlin wrote: > The Transaction should be unaware of its role in the ActiveEntry. Just check if this Transaction is read-only. Done
Patchset #14 (id:320001) has been deleted
Also renamed state STATE_WAIT_BEFORE_READ to STATE_FINISH_HEADERS as discussed since it is also now reached when a partial request is actually "reading_" and goes back to the headers state machine. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1852: if (entry_->headers_transaction == this) On 2017/03/30 at 16:38:41, shivanisha wrote: > On 2017/03/30 at 14:25:26, jkarlin wrote: > > The Transaction should be unaware of its role in the ActiveEntry. Just check if this Transaction is read-only. > We cannot check mode to be read-only to imply that it is currently a reader since mode is also set to read when we want it to be a future reader. Invoked DoneResponseHeaders for readers as well and checked it there. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1867: On 2017/03/30 at 14:25:26, jkarlin wrote: > It looks like at this point all paths lead to return OK. One path leads to mode_ = READ and resetting the network transaction. Perhaps instead the HttpCache should force the Transaction into READ mode so that it doesn't have to infer its place in the ActiveEntry? Done. Added a function ConvertToReadMode to be invoked from HttpCache when inserting the transaction in readers.
Love it! https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.h#... net/http/http_cache.h:401: int DoneResponseHeaders(ActiveEntry* entry, Suggest DoneWithResponseHeaders https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:610: // BeginCacheValidation() -> SetupEntryForRead() -> WaitBeforeRead* Need to update these comments for the new state name and the fact that we always go to STATE_FINISH_HEADERS. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1833: if (auth_response_.headers.get() || Do we need this check? I'd rather we went through DoneResponseHeaders if possible. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1841: return OK; Do we need this check? I'd rather we went through DoneResponseHeaders if possible. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1869: mode_ = READ; DCHECK that we're in a safe state to call ConvertToReadMode.
My apologies that this has taken so long; this week's been both busy and stressful. These are just http_cache.*; I'll try to get to HC::T later today. When you have a chance (preferably > 4pm Today (Thursday)) I'd like to talk through the io callback issues with you--there's something at the conceptual level I'm not getting about those. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:965: ProcessQueuedTransactions(entry); Is it reasonable to avoid another thread hop here and just loop? https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:969: entry->add_to_entry_queue.erase(entry->add_to_entry_queue.begin()); Would it be reasonable to rewrite the above so that this is more clearly associated with the decision to actually move the transaction into a different place in the active entry? I'm thinking of something like: Transaction* transaction = entry->add_to_entry_queue.front(); if (entry->writer && !(transaction->mode() & Transaction::WRITE)) return entry->add_to_entry_queue.erase(transaction); // Or use begin() if you want, as long as it's clear by proxy that you're talking about the same transaction. if (transaction->mode() & Transaction::WRITE) entry->headers_transaction = transaction; else { entry->readers.insert(transaction); ProcessQueuedTransacations(); } ? https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:981: // Response body is already written, convert to a reader. Isn't the response body about the state of the ActiveEntry rather than the transaction being a READ one? READ transactions can occur based on LOAD_ONLY_FROM_CACHE, I think. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:987: ProcessQueuedTransactions(entry); Given that we're already a PostTask away from the main thread, would it be possible to loop instead? https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:1032: if (!entry->done_headers_queue.empty()) { I think you can leave the outer conditional off without any performance cost and with a slight decrease in code complexity? https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:1083: // If writer fails, all transactions should fail. All transactions? This is just failing the headers_transaction; the ones in the queues should not fail? https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.h#... net/http/http_cache.h:414: const std::string& method); const? (More interested for documentation purposes than anything else.) https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.cc... net/http/http_cache.cc:936: return; Why return? Is there some reason not to process the add to entry queue after we've processed the done headers queue? https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.cc... net/http/http_cache.cc:1084: Transaction* transaction) { nit, not even a suggestion (partially because looking at the code it looks like a hassle): I don't like the interface where a full pointer is being passed in just to make the writer vs. headers_transaction choice. My preference would actually be to have two interfaces (DoneValidatingHeaders/DoneWritingToEntry) and not pass in the transaction. But that looks like it would take some refactoring in HC::T, so don't do it unless the idea inspires you :-}. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.h#... net/http/http_cache.h:452: void ProcessAddToEntryTransaction(ActiveEntry* entry); nit, suggestion: This is more processing the queue than processing a transaction. ProcessEntryQueue? Same comment on ProcessDoneHeadersTransaction below.
Most recent feedback from all reviewers addressed. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:965: ProcessQueuedTransactions(entry); On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > Is it reasonable to avoid another thread hop here and just loop? Looping here will mean that the control flow goes in more than one transactions io callbacks in sequence. Posting a task makes it clear that everytime it will be just one transaction's io callback invoked. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:969: entry->add_to_entry_queue.erase(entry->add_to_entry_queue.begin()); On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > Would it be reasonable to rewrite the above so that this is more clearly associated with the decision to actually move the transaction into a different place in the active entry? I'm thinking of something like: > > Transaction* transaction = entry->add_to_entry_queue.front(); > if (entry->writer && !(transaction->mode() & Transaction::WRITE)) > return > entry->add_to_entry_queue.erase(transaction); // Or use begin() if you want, as long as it's clear by proxy that you're talking about the same transaction. > > if (transaction->mode() & Transaction::WRITE) > entry->headers_transaction = transaction; > else { > entry->readers.insert(transaction); ProcessQueuedTransacations(); > } > > ? Done. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:981: // Response body is already written, convert to a reader. On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > Isn't the response body about the state of the ActiveEntry rather than the transaction being a READ one? READ transactions can occur based on LOAD_ONLY_FROM_CACHE, I think. This comment here signifies that since we are in ProcessDoneHeadersTransaction, response body must have been complete. A transaction will be here with Read mode not because of LOAD_ONLY_FROM_CACHE but because its validation led to a match. (For read-only transactions because of LOAD_ONLY_FROM_CACHE, their states are from add_to_entry_queue to readers.) https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:987: ProcessQueuedTransactions(entry); On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > Given that we're already a PostTask away from the main thread, would it be possible to loop instead? Looping here will mean that the control flow goes in more than one transactions io callbacks in sequence. Posting a task makes it clear that everytime it will be just one transaction's io callback invoked. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:992: bool HttpCache::IsCancelResponseBody(ActiveEntry* entry, On 2017/03/30 at 14:58:14, jkarlin wrote: > The name IsCancelResponseBody doesn't make sense to me. > > Perhaps IsBackendEntryFinished? Actually, what this function needs to return is this transaction's role in the entry, how about: IsTransactionCurrentOrFutureWriter()? https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:999: On 2017/03/30 at 14:58:14, jkarlin wrote: > It seems like this function shouldn't be necessary. Can the Transaction really not tell: > > 1) If it has written bytes to the cache entry > 2) If it isn't yet done writing bytes to the cache entry Let's see... a transaction can tell above 1) and 2) but if 1) is not true, how does it tell if it was the one responsible for writing the entry , that is done by checking if the transaction's mode is WRITE and there is no writer currently in the entry. Simplified and renamed the function, PTAL. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:1032: if (!entry->done_headers_queue.empty()) { On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > I think you can leave the outer conditional off without any performance cost and with a slight decrease in code complexity? Removed. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:1083: // If writer fails, all transactions should fail. On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > All transactions? This is just failing the headers_transaction; the ones in the queues should not fail? Changed the comment to : "... all transactions should be restarted" https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.h#... net/http/http_cache.h:414: const std::string& method); On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > const? (More interested for documentation purposes than anything else.) Done. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.cc... net/http/http_cache.cc:936: return; On 2017/03/30 at 19:18:06, Randy Smith (Not in Mondays) wrote: > Why return? Is there some reason not to process the add to entry queue after we've processed the done headers queue? Yes, we do not want to process multiple transactions' io_callback in the same flow. If needed, it will be processed in a separate task posted from inside ProcessDoneHeadersTransaction. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.cc... net/http/http_cache.cc:1084: Transaction* transaction) { On 2017/03/30 at 19:18:06, Randy Smith (Not in Mondays) wrote: > nit, not even a suggestion (partially because looking at the code it looks like a hassle): I don't like the interface where a full pointer is being passed in just to make the writer vs. headers_transaction choice. My preference would actually be to have two interfaces (DoneValidatingHeaders/DoneWritingToEntry) and not pass in the transaction. But that looks like it would take some refactoring in HC::T, so don't do it unless the idea inspires you :-}. I kind of like it that HCT does not have to decide which function to invoke based on some conditions. Note that it is possible that HCT is in the headers state machine but a writer (in range requests case) https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.h#... net/http/http_cache.h:401: int DoneResponseHeaders(ActiveEntry* entry, On 2017/03/30 at 17:04:54, jkarlin wrote: > Suggest DoneWithResponseHeaders Done. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache.h#... net/http/http_cache.h:452: void ProcessAddToEntryTransaction(ActiveEntry* entry); On 2017/03/30 at 19:18:06, Randy Smith (Not in Mondays) wrote: > nit, suggestion: This is more processing the queue than processing a transaction. ProcessEntryQueue? Same comment on ProcessDoneHeadersTransaction below. Done. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:610: // BeginCacheValidation() -> SetupEntryForRead() -> WaitBeforeRead* On 2017/03/30 at 17:04:54, jkarlin wrote: > Need to update these comments for the new state name and the fact that we always go to STATE_FINISH_HEADERS. Updated the comments to rename and include the state in all flows. Also traversed through all the start state machine functions and now they are exiting in state FINISH_HEADERS instead of NONE even in error scenarios or scenarios where entry_ is not set. And then there is early exit from DoFinishHeaders when result != 0 or !entry_. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1833: if (auth_response_.headers.get() || On 2017/03/30 at 17:04:54, jkarlin wrote: > Do we need this check? I'd rather we went through DoneResponseHeaders if possible. This check is to do with the fact that this transaction may be restarted by the consumer using the RestartWithAuth related functions. I think its ok to check it here since its this transaction's state that we are checking in auth_response_ and not the entry's state. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1841: return OK; On 2017/03/30 at 17:04:54, jkarlin wrote: > Do we need this check? I'd rather we went through DoneResponseHeaders if possible. We need this check because these requests should not go past headers_transaction state. Here also , I feel the check is not related to ActiveEntry's state so its ok to make it here. https://codereview.chromium.org/2721933002/diff/340001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1869: mode_ = READ; On 2017/03/30 at 17:04:54, jkarlin wrote: > DCHECK that we're in a safe state to call ConvertToReadMode. Added DCHECK(!reading_)
Next round of comments. This is close, which is really awesome! Some notes on the below comments: * The question of looping rather than PostTasking is something I care about, but not something that you're changing, so I won't block the CL on it. Having said that, it seems wrong to me, and I'd like to at least get to a point where we share a perspective :-}. * Feel free to grab me for a conversation on any of the conceptual questions (any of them, really, but the conceptual questions are the ones where I could imagine interaction being most useful). I'm counting those as ConvertToReadMode() effect, done_headers_queue.front(), and the relationship between ERR_CACHE_RACE and SetValidatingCannotProceed(). Thanks! https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:965: ProcessQueuedTransactions(entry); On 2017/03/31 17:47:05, shivanisha wrote: > On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > > Is it reasonable to avoid another thread hop here and just loop? > > Looping here will mean that the control flow goes in more than one transactions > io callbacks in sequence. Posting a task makes it clear that everytime it will > be just one transaction's io callback invoked. Why is that a useful invariant? https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:1083: // If writer fails, all transactions should fail. On 2017/03/31 17:47:05, shivanisha wrote: > On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > > All transactions? This is just failing the headers_transaction; the ones in > the queues should not fail? > > Changed the comment to : "... all transactions should be restarted" Still confused, though maybe it's just about placement. The restart happens in ProcessEntryFailure, correct? So shouldn't this comment be above that call? What's happening in this conditional is about failing the entry->headers_transaction, not the other transactions. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc... net/http/http_cache.cc:972: transaction->ConvertToReadMode(); What transition in the transaction mode is actually happening here? Specifically, looking at the definition of HC::T::Mode, I see three bits: 0 is READ_META, 1 is READ_DATA, and 2 is WIRTE. Here and below we're only converting if WRITE isn't set, and READ == READ_META | READ_DATA. So it seems as if the implication of ConvertToReadMode() is wrong; either we aren't changing the mode at all, or we're upgrading it (we had zero or one of READ_META & READ_DATA, and we've moving to having both). Is that correct? Is there a better name for this function if it is? https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc... net/http/http_cache.cc:1020: if (transaction == entry->done_headers_queue.front()) Why is the check against the front of the done_headers queue rather than for any place in the queue? https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:582: void HttpCache::Transaction::SetValidatingCannotProceed() { Suggestion: As I understand it, this is only expected to be called while we're in header validation. Should we DCHECK that? https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.h:118: const CompletionCallback& io_callback() { return io_callback_; } Would you be willing to put a documentation comment here as we discussed a bit in person? The key thing I'd like called out is that, at least for HC<->HC::T communciation, this *isn't* the usual IO Callback (which generally means "Whatever you (the consumer class) requested is done and I have no further knowledge of you". The first part is true ("Operation requested is done") and part of the second is true ("Nothing further will happen until you get back to me") but the HC::T is still tied to the HC and needs to take that into account. Possibly the right place for this is near OnIOComplete(), and should also acknowledge that this callback is used for disk writes and network reads as well (which I think do behave like the regular io completions). https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.h:169: void SetValidatingCannotProceed(); So conceptually this seems like the same thing as ERR_CACHE_RACE, except that ERR_CACHE_RACE is sent synchronously and this can happen asynchronously. Is there any way to combine both signals into one? If not, I'd at least like to make the connection (and difference) clear in the comments. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.h:259: int DoLoopImpl(int result); I don't think this exists anymore?
https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc... net/http/http_cache.cc:1073: // Transaction is done writing to entry in the response body phase. Does this comment only apply to the transaction == entry->writer case? If so, please put the comment inside that. Right now it also looks like it applies to the else. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:376: // should not be allowed in the Read() method. Fix those tests. I assume that the tests are defensive due to the high likelihood that a customer would call READ() on a HEAD requestion. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1596: (entry_->writer || !entry_->readers.empty())) { In regards to: entry_->writer || !entry_->readers.empty() Perhaps instead, cache_->CanOverwriteActiveEntry()? That way the HCT isn't aware of the ActiveEntry's states. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1597: DCHECK_EQ(entry_->headers_transaction, this); Why not call DoneWritingToEntry(false); here like on line 1540 instead of calling cache_->DoneWithResponseHeaders?
Feedback addressed and added some ActiveEntry's state related details in the test framework. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:965: ProcessQueuedTransactions(entry); On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > On 2017/03/31 17:47:05, shivanisha wrote: > > On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > > > Is it reasonable to avoid another thread hop here and just loop? > > > > Looping here will mean that the control flow goes in more than one transactions > > io callbacks in sequence. Posting a task makes it clear that everytime it will > > be just one transaction's io callback invoked. > > Why is that a useful invariant? As discussed f2f, I have preferred to keep the post task to make one call stack correspond to one transaction as it has been till now, but I do not see a problem why we cannot have a loop here especially since entry should not get deleted before the loop is entered a second time. Added a todo as well. https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:995: bool cancel = (method != "HEAD"); On 2017/03/30 at 14:58:14, jkarlin wrote: > The cancel variable is unnecessary and adds confusion. The following works just as well: > > if (method == "HEAD") > return false; > > if (transaction == entry->writer) > return true; > > ... done https://codereview.chromium.org/2721933002/diff/300001/net/http/http_cache.cc... net/http/http_cache.cc:1083: // If writer fails, all transactions should fail. On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > On 2017/03/31 17:47:05, shivanisha wrote: > > On 2017/03/30 at 19:18:05, Randy Smith (Not in Mondays) wrote: > > > All transactions? This is just failing the headers_transaction; the ones in > > the queues should not fail? > > > > Changed the comment to : "... all transactions should be restarted" > > Still confused, though maybe it's just about placement. The restart happens in ProcessEntryFailure, correct? So shouldn't this comment be above that call? What's happening in this conditional is about failing the entry->headers_transaction, not the other transactions. Ah I see, changed the comment to just mention headers_transaction. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc... net/http/http_cache.cc:972: transaction->ConvertToReadMode(); On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > What transition in the transaction mode is actually happening here? Specifically, looking at the definition of HC::T::Mode, I see three bits: 0 is READ_META, 1 is READ_DATA, and 2 is WIRTE. Here and below we're only converting if WRITE isn't set, and READ == READ_META | READ_DATA. So it seems as if the implication of ConvertToReadMode() is wrong; either we aren't changing the mode at all, or we're upgrading it (we had zero or one of READ_META & READ_DATA, and we've moving to having both). Is that correct? Is there a better name for this function if it is? Thanks for pointing this out. We are actually converting the mode_ to READ and resetting network transaction in all the places where validation was a match before we come here. So we do not need this function at all. This logic was a remnant from the initial patches where there was a conditional ConvertWriterToReader. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc... net/http/http_cache.cc:1020: if (transaction == entry->done_headers_queue.front()) On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > Why is the check against the front of the done_headers queue rather than for any place in the queue? The invariant is that the first transaction that created the entry will go from headers_transaction -> done_headers_queue -> writer and will write the response. Any later transactions added to done_headers_queue should actually be a result of validation match and thus have mode as READ. Added a comment to that effect. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc... net/http/http_cache.cc:1073: // Transaction is done writing to entry in the response body phase. On 2017/04/03 at 14:32:48, jkarlin wrote: > Does this comment only apply to the transaction == entry->writer case? If so, please put the comment inside that. Right now it also looks like it applies to the else. Removed the comment since it is clear from the condition. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:376: // should not be allowed in the Read() method. Fix those tests. On 2017/04/03 at 14:32:48, jkarlin wrote: > I assume that the tests are defensive due to the high likelihood that a customer would call READ() on a HEAD requestion. That might be the case, removing the todo from here. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:582: void HttpCache::Transaction::SetValidatingCannotProceed() { On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > Suggestion: As I understand it, this is only expected to be called while we're in header validation. Should we DCHECK that? Done. Added DCHECK(!reading_) https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1596: (entry_->writer || !entry_->readers.empty())) { On 2017/04/03 at 14:32:48, jkarlin wrote: > In regards to: entry_->writer || !entry_->readers.empty() > > Perhaps instead, cache_->CanOverwriteActiveEntry()? > > That way the HCT isn't aware of the ActiveEntry's states. Good call, invoked IsCurrentOrFutureWriter for this case as well. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1597: DCHECK_EQ(entry_->headers_transaction, this); On 2017/04/03 at 14:32:48, jkarlin wrote: > Why not call DoneWritingToEntry(false); here like on line 1540 instead of calling cache_->DoneWithResponseHeaders? Done and removed is_match argument from DoneWithResponseHeaders https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.h:118: const CompletionCallback& io_callback() { return io_callback_; } On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > Would you be willing to put a documentation comment here as we discussed a bit in person? The key thing I'd like called out is that, at least for HC<->HC::T communciation, this *isn't* the usual IO Callback (which generally means "Whatever you (the consumer class) requested is done and I have no further knowledge of you". The first part is true ("Operation requested is done") and part of the second is true ("Nothing further will happen until you get back to me") but the HC::T is still tied to the HC and needs to take that into account. > > Possibly the right place for this is near OnIOComplete(), and should also acknowledge that this callback is used for disk writes and network reads as well (which I think do behave like the regular io completions). Done. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.h:169: void SetValidatingCannotProceed(); On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > So conceptually this seems like the same thing as ERR_CACHE_RACE, except that ERR_CACHE_RACE is sent synchronously and this can happen asynchronously. Is there any way to combine both signals into one? > > If not, I'd at least like to make the connection (and difference) clear in the comments. Combined the 2 in the following manner: - Created a separate state STATE_HEADERS_PHASE_CANNOT_PROCEED which will be set in this function instead of a boolean so no extra code will be needed in OnIOComplete(). - In all ERR_CACHE_RACE scenarios also, the same state will be set. - The new state's Do function will be responsible for restarting the state machine to STATE_INIT_ENTRY and other variables' resetting. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.h:259: int DoLoopImpl(int result); On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > I don't think this exists anymore? Removed
https://codereview.chromium.org/2721933002/diff/380001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/380001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1598: !cache_->IsTransactionCurrentOrFutureWriter(entry_, this, I don't think the HCT should be aware of ActiveEntry states (e.g., writers, readers, or pending headers). I'm trying to avoid a combinatorial explosion of states (e.g., having to think about all 3 ActiveEntry states for each HCT state). As such, HC::IsTransactionCurrentOrFutureWriter is too specific. I'd prefer something like, "HC::CanTransactionWriteToEntry". That function should be called before first updating/overwriting headers or first writing body bytes and it should be DCHECKED before each call to entry_->disk_entry->WriteData. WDYT? Would something like that work?
Tests added.
Description was changed from ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. More tests are coming. BUG=472740 ========== to ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. BUG=472740 ==========
https://codereview.chromium.org/2721933002/diff/380001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/380001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1598: !cache_->IsTransactionCurrentOrFutureWriter(entry_, this, On 2017/04/04 at 16:49:55, jkarlin wrote: > I don't think the HCT should be aware of ActiveEntry states (e.g., writers, readers, or pending headers). I'm trying to avoid a combinatorial explosion of states (e.g., having to think about all 3 ActiveEntry states for each HCT state). > > As such, HC::IsTransactionCurrentOrFutureWriter is too specific. I'd prefer something like, "HC::CanTransactionWriteToEntry". That function should be called before first updating/overwriting headers or first writing body bytes and it should be DCHECKED before each call to entry_->disk_entry->WriteData. > > WDYT? Would something like that work? Created CanTransactionWriteResponseHeaders for this scenario and it will not be invoked here but in DoCacheWriteResponse before attempting to write headers to the entry. Renamed IsTransactionCurrentOrFutureWriter to CanTransactionWriteResponseBody to be invoked in the destructor as it is done now and invoked with a DCHECK in WriteToEntry everytime it is about to write data to entry.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: ios-device on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device/builds...) ios-device-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device-xcode-...) ios-simulator on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/bui...) ios-simulator-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator-xco...) mac_chromium_compile_dbg_ng on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_comp...)
Looking good. https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc... net/http/http_cache.cc:1020: if (transaction == entry->done_headers_queue.front()) On 2017/04/03 20:16:48, shivanisha wrote: > On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > > Why is the check against the front of the done_headers queue rather than for > any place in the queue? > > The invariant is that the first transaction that created the entry will go from > headers_transaction -> done_headers_queue -> writer and will write the response. > Any later transactions added to done_headers_queue should actually be a result > of validation match and thus have mode as READ. Added a comment to that effect. So does that mean that if the writer completes and we have only readers, this function might return true for a late arriving reader? https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache_tr... net/http/http_cache_transaction.h:169: void SetValidatingCannotProceed(); On 2017/04/03 20:16:49, shivanisha wrote: > On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > > So conceptually this seems like the same thing as ERR_CACHE_RACE, except that > ERR_CACHE_RACE is sent synchronously and this can happen asynchronously. Is > there any way to combine both signals into one? > > > > If not, I'd at least like to make the connection (and difference) clear in the > comments. > > Combined the 2 in the following manner: > - Created a separate state STATE_HEADERS_PHASE_CANNOT_PROCEED which will be set > in this function instead of a boolean so no extra code will be needed in > OnIOComplete(). > - In all ERR_CACHE_RACE scenarios also, the same state will be set. > - The new state's Do function will be responsible for restarting the state > machine to STATE_INIT_ENTRY and other variables' resetting. Awesome! Thank you. https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h#... net/http/http_cache.h:413: // writing the response body. I feel as if the comments and the name don't quite match. Is this about whether the transaction can write the response body at some point in the future, or is it about whether it's committed to writing the response body? https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... File net/http/http_cache_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1557: // any impact to other transactions. What if the currently active headers transaction is destroyed (not just one that's completed validation)? What happens to the rest of the entry? Should we have a test for that? (This may require interfering at the network level to pause the network transaction, so you can leave the headers transaction in a state where it's waiting for the network.) https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1756: } Should we also confirm that destroying a reader doesn't affect either an active headers transaction or transactions in either the add_to_entry queue or the done_headers queue?
https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/360001/net/http/http_cache.cc... net/http/http_cache.cc:1020: if (transaction == entry->done_headers_queue.front()) On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > On 2017/04/03 20:16:48, shivanisha wrote: > > On 2017/03/31 at 19:26:12, Randy Smith (Not in Mondays) wrote: > > > Why is the check against the front of the done_headers queue rather than for > > any place in the queue? > > > > The invariant is that the first transaction that created the entry will go from > > headers_transaction -> done_headers_queue -> writer and will write the response. > > Any later transactions added to done_headers_queue should actually be a result > > of validation match and thus have mode as READ. Added a comment to that effect. > > So does that mean that if the writer completes and we have only readers, this function might return true for a late arriving reader? The check on mode takes care of a reader always returning false. Also, the latest patch updates the transaction == entry->headers_transaction condition to return !entry->writer && entry->done_headers_queue.empty(); https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h#... net/http/http_cache.h:413: // writing the response body. On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > I feel as if the comments and the name don't quite match. Is this about whether the transaction can write the response body at some point in the future, or is it about whether it's committed to writing the response body? Since there can be only one writer, "can" implies the transaction is committed to write. Let me know if you think the name or comments should change. https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... File net/http/http_cache_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1557: // any impact to other transactions. On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > What if the currently active headers transaction is destroyed (not just one that's completed validation)? What happens to the rest of the entry? Should we have a test for that? > > (This may require interfering at the network level to pause the network transaction, so you can leave the headers transaction in a state where it's waiting for the network.) It seems there doesn't exist a way to pause and resume the mock network transaction currently. Will try to add that functionality in the MockNetworkTransaction to create such a test, in a subsequent patch. https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1756: } On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > Should we also confirm that destroying a reader doesn't affect either an active headers transaction or transactions in either the add_to_entry queue or the done_headers queue? Added a test for that: HttpCache.SimpleGET_ParallelValidationCancelReader Could not test the impact on headers_transaction for the same reason as cited above about not being able to pause the network transaction. Will add that once I have that framework. Also when one transaction becomes a reader, all in done_headers_queue will also become readers so can't check the impact of cancelling the reader on a transaction in done_headers_queue.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: chromium_presubmit on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...) linux_chromium_compile_dbg_ng on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...) linux_chromium_tsan_rel_ng on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...) ios-device-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device-xcode-...) ios-simulator on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/bui...) mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Patchset #20 (id:460001) has been deleted
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: mac_chromium_compile_dbg_ng on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_comp...) mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...) win_clang on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_clang/builds/...)
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Patchset #20 (id:480001) has been deleted
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: mac_chromium_compile_dbg_ng on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_comp...)
I'm good with this CL except for the two tests I suggested. If you really want to delay them into another CL, I'm ok with that (so LGTM), but as noted below I think the current infrastructure would support delaying a network request. https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h#... net/http/http_cache.h:413: // writing the response body. On 2017/04/05 15:00:37, shivanisha wrote: > On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > > I feel as if the comments and the name don't quite match. Is this about > whether the transaction can write the response body at some point in the future, > or is it about whether it's committed to writing the response body? > > Since there can be only one writer, "can" implies the transaction is committed > to write. Let me know if you think the name or comments should change. I have a preference for a name that indicates that the transaction will be writing the response body, but up to you. https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... File net/http/http_cache_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1557: // any impact to other transactions. On 2017/04/05 15:00:37, shivanisha wrote: > On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > > What if the currently active headers transaction is destroyed (not just one > that's completed validation)? What happens to the rest of the entry? Should we > have a test for that? > > > > (This may require interfering at the network level to pause the network > transaction, so you can leave the headers transaction in a state where it's > waiting for the network.) > > It seems there doesn't exist a way to pause and resume the mock network > transaction currently. Will try to add that functionality in the > MockNetworkTransaction to create such a test, in a subsequent patch. I'm not sure that's true. Did you look at the BeforeNetworkStartCallback on the MockNetworkTransaction? It looks like it has a defer argument that could be used to pause a network callback. https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1756: } On 2017/04/05 15:00:37, shivanisha wrote: > On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > > Should we also confirm that destroying a reader doesn't affect either an > active headers transaction or transactions in either the add_to_entry queue or > the done_headers queue? > > Added a test for that: HttpCache.SimpleGET_ParallelValidationCancelReader > > Could not test the impact on headers_transaction for the same reason as cited > above about not being able to pause the network transaction. Will add that once > I have that framework. > Also when one transaction becomes a reader, all in done_headers_queue will also > become readers so can't check the impact of cancelling the reader on a > transaction in done_headers_queue. Acknowledged.
On 2017/04/05 at 18:39:18, rdsmith wrote: > I'm good with this CL except for the two tests I suggested. If you really want to delay them into another CL, I'm ok with that (so LGTM), but as noted below I think the current infrastructure would support delaying a network request. > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h > File net/http/http_cache.h (right): > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h#... > net/http/http_cache.h:413: // writing the response body. > On 2017/04/05 15:00:37, shivanisha wrote: > > On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > > > I feel as if the comments and the name don't quite match. Is this about > > whether the transaction can write the response body at some point in the future, > > or is it about whether it's committed to writing the response body? > > > > Since there can be only one writer, "can" implies the transaction is committed > > to write. Let me know if you think the name or comments should change. > > I have a preference for a name that indicates that the transaction will be writing the response body, but up to you. > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... > File net/http/http_cache_unittest.cc (right): > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... > net/http/http_cache_unittest.cc:1557: // any impact to other transactions. > On 2017/04/05 15:00:37, shivanisha wrote: > > On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > > > What if the currently active headers transaction is destroyed (not just one > > that's completed validation)? What happens to the rest of the entry? Should we > > have a test for that? > > > > > > (This may require interfering at the network level to pause the network > > transaction, so you can leave the headers transaction in a state where it's > > waiting for the network.) > > > > It seems there doesn't exist a way to pause and resume the mock network > > transaction currently. Will try to add that functionality in the > > MockNetworkTransaction to create such a test, in a subsequent patch. > > I'm not sure that's true. Did you look at the BeforeNetworkStartCallback on the MockNetworkTransaction? It looks like it has a defer argument that could be used to pause a network callback. > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... > net/http/http_cache_unittest.cc:1756: } > On 2017/04/05 15:00:37, shivanisha wrote: > > On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > > > Should we also confirm that destroying a reader doesn't affect either an > > active headers transaction or transactions in either the add_to_entry queue or > > the done_headers queue? > > > > Added a test for that: HttpCache.SimpleGET_ParallelValidationCancelReader > > > > Could not test the impact on headers_transaction for the same reason as cited > > above about not being able to pause the network transaction. Will add that once > > I have that framework. > > Also when one transaction becomes a reader, all in done_headers_queue will also > > become readers so can't check the impact of cancelling the reader on a > > transaction in done_headers_queue. > > Acknowledged. Just clarifying, I intend on adding the headers_transaction related tests in the same CL (but a subsequent patch).
On 2017/04/05 18:49:58, shivanisha wrote: > On 2017/04/05 at 18:39:18, rdsmith wrote: > > I'm good with this CL except for the two tests I suggested. If you really > want to delay them into another CL, I'm ok with that (so LGTM), but as noted > below I think the current infrastructure would support delaying a network > request. > > > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h > > File net/http/http_cache.h (right): > > > > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache.h#... > > net/http/http_cache.h:413: // writing the response body. > > On 2017/04/05 15:00:37, shivanisha wrote: > > > On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > > > > I feel as if the comments and the name don't quite match. Is this about > > > whether the transaction can write the response body at some point in the > future, > > > or is it about whether it's committed to writing the response body? > > > > > > Since there can be only one writer, "can" implies the transaction is > committed > > > to write. Let me know if you think the name or comments should change. > > > > I have a preference for a name that indicates that the transaction will be > writing the response body, but up to you. > > > > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... > > File net/http/http_cache_unittest.cc (right): > > > > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... > > net/http/http_cache_unittest.cc:1557: // any impact to other transactions. > > On 2017/04/05 15:00:37, shivanisha wrote: > > > On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > > > > What if the currently active headers transaction is destroyed (not just > one > > > that's completed validation)? What happens to the rest of the entry? > Should we > > > have a test for that? > > > > > > > > (This may require interfering at the network level to pause the network > > > transaction, so you can leave the headers transaction in a state where it's > > > waiting for the network.) > > > > > > It seems there doesn't exist a way to pause and resume the mock network > > > transaction currently. Will try to add that functionality in the > > > MockNetworkTransaction to create such a test, in a subsequent patch. > > > > I'm not sure that's true. Did you look at the BeforeNetworkStartCallback on > the MockNetworkTransaction? It looks like it has a defer argument that could be > used to pause a network callback. > > > > > https://codereview.chromium.org/2721933002/diff/420001/net/http/http_cache_un... > > net/http/http_cache_unittest.cc:1756: } > > On 2017/04/05 15:00:37, shivanisha wrote: > > > On 2017/04/04 at 21:21:16, Randy Smith (Not in Mondays) wrote: > > > > Should we also confirm that destroying a reader doesn't affect either an > > > active headers transaction or transactions in either the add_to_entry queue > or > > > the done_headers queue? > > > > > > Added a test for that: HttpCache.SimpleGET_ParallelValidationCancelReader > > > > > > Could not test the impact on headers_transaction for the same reason as > cited > > > above about not being able to pause the network transaction. Will add that > once > > > I have that framework. > > > Also when one transaction becomes a reader, all in done_headers_queue will > also > > > become readers so can't check the impact of cancelling the reader on a > > > transaction in done_headers_queue. > > > > Acknowledged. > > Just clarifying, I intend on adding the headers_transaction related tests in the > same CL (but a subsequent patch). Whoops, sorry. That sounds good; thanks!
First pass at http_cache.h/cc. Looks good. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:855: return OK; Once read-only transactions follow the normal path this condition can go right? https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:856: DCHECK_EQ(entry->headers_transaction, transaction); https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:930: return; If you posttask the io callbacks (as mentioned in comments below) then this return should go. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:937: void HttpCache::ProcessAddToEntryQueue(ActiveEntry* entry) { Can you add a comment above this function definition explaining what this function is trying to accomplish? https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:942: // If transaction is read-only and entry is not yet complete, then it Can remove the read-only comment. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:952: return; This condition can go once we stop special-casing read-only right? https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:961: // A read-only transaction can start reading the response. This condition can also go. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:988: // TODO (shivanisha@) Consider a loop instead of posting a task. I assume the concern with looping is reentrancy? If you're willing to run one transaction's io_callback synchronously are there really any extra issues with running multiple? Might the first one change the cache's state somehow? Alternatively, if you're still concerned, just PostTask the call to io_callback and process them all in a loop. That way each of the queues is processed in one function call. If you do that, then don't PostTask the call to OnProcessQueuedTransactions as it's unnecessary. Same goes for ProcessAddToEntryQueue. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:1023: transaction->mode() == Transaction::NONE)) I think you want: if ( !(transaction>mode() & Transaction::WRITE) || transaction->mode() == Transaction == NONE ) right? In which case, you can simplify that to: if (!transaction->mode() & Transaction::WRITE). https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:1027: return !entry->writer && entry->done_headers_queue.empty(); I don't understand this conditional. Can you add a comment? https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:1041: auto i = std::find(entry->done_headers_queue.begin(), s/i/it/ https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:1104: auto it = entry->readers.find(transaction); Can you add a TODO that this is a linear operation and it would be good to make it faster? https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:268: // We implement a basic reader/writer lock for the disk cache entry. If Pull this comment up above the definition of the ActiveEntry class. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:279: // add_to_entry_queue-> headers_transaction -> done_headers_queue -> I'd love to see the state machine description above the definition of ActiveEntry as well. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:285: // add_to_entry_queue-> readers (once the data is written to the cache by Why special case read-only transactions? It seems like it should follow the normal path of add_to_entry_queue-> headers_transaction-> done_headers_queue-> readers. After F2F we're in agreement about this. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:393: // and the transaction will be notified about completion via its IO callback. Comment on what error to expect on failure. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:399: // to process queued transactions of the entry. Returns ERR_IO_PENDING. Add "the transaction will be notified about completion via its IO callback" as in the comment for AddTransactionToEntry https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:399: // to process queued transactions of the entry. Returns ERR_IO_PENDING. Comment on what error to expect on failure. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:413: // writing the response body. Given the above method, and the name of this method, I'd expect the comment to say: "Returns true if this transaction can write response body bytes to the entry." Instead it's really a method that tries to answer two questions: 1) Can the transaction write to the response body? 2) Did the transaction start writing to this entry but not yet finish? I believe we can split those into two functions: 1) CanTransactionWriteResponseBody 2) IsTransactionWritingIncomplete WDYT? https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:446: void RemoveAllQueuedTransactions(ActiveEntry* entry, TransactionList* list); Note that these methods need to appear in cc file in the same order. Right now ProcessEntryFailure appears above ProcessQueuedTransactions. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:200: bool cancel_response = cache_->CanTransactionWriteResponseBody( nit: s/cancel_response/writing_incomplete/ https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1577: nit: remove newline https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1710: // cannot write to this entry. Also mention that the transaction continues on without writing to the backend. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1841: return cache_->DoneWithResponseHeaders(entry_, this); Since you're not passing a callback it's non-obvious that this is an async call. Can you add a comment that the cache will run the io_callback_?
Added a patch that includes the framework for pausing and restarting a mock network transaction. The framework is based on BeforeNetworkStartCallback which was not completely implemented in MockNetworkTransaction. Based on the framework: - Added a new test SimpleGET_ParallelValidationCancelHeaders - Modified SimpleGET_ParallelValidationCancelReader to see headers_transaction should not be impacted due to reader deletion. - Modified SimpleGET_ParallelValidationCancelWriter to see impact of deleting writer on headers_transaction.
Patchset #22 (id:540001) has been deleted
rdsmith@, PTAL if you would like for tests added for headers_transaction state. Also, I ended up adding a loop instead of posting a task during ProcessDoneHeadersTransaction. Simplification of read-only transactions' state transitions to be similar to others helped me take this decision, since making the loop was a very little change now (in just one function). jkarlin@, PTAL for latest feedback addressed. Thanks! https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:855: return OK; On 2017/04/06 at 17:37:13, jkarlin wrote: > Once read-only transactions follow the normal path this condition can go right? Yes, converted to DCHECK_EQ(entry->headers_transaction, transaction) https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:856: On 2017/04/06 at 17:37:13, jkarlin wrote: > DCHECK_EQ(entry->headers_transaction, transaction); done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:930: return; On 2017/04/06 at 17:37:13, jkarlin wrote: > If you posttask the io callbacks (as mentioned in comments below) then this return should go. Removed the return https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:937: void HttpCache::ProcessAddToEntryQueue(ActiveEntry* entry) { On 2017/04/06 at 17:37:13, jkarlin wrote: > Can you add a comment above this function definition explaining what this function is trying to accomplish? done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:942: // If transaction is read-only and entry is not yet complete, then it On 2017/04/06 at 17:37:13, jkarlin wrote: > Can remove the read-only comment. done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:952: return; On 2017/04/06 at 17:37:13, jkarlin wrote: > This condition can go once we stop special-casing read-only right? Yes, removed https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:961: // A read-only transaction can start reading the response. On 2017/04/06 at 17:37:13, jkarlin wrote: > This condition can also go. Removed https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:988: // TODO (shivanisha@) Consider a loop instead of posting a task. On 2017/04/06 at 17:37:13, jkarlin wrote: > I assume the concern with looping is reentrancy? > > If you're willing to run one transaction's io_callback synchronously are there really any extra issues with running multiple? Might the first one change the cache's state somehow? > > Alternatively, if you're still concerned, just PostTask the call to io_callback and process them all in a loop. That way each of the queues is processed in one function call. If you do that, then don't PostTask the call to OnProcessQueuedTransactions as it's unnecessary. > > Same goes for ProcessAddToEntryQueue. Introduced a loop instead of posting a task. As discussed earlier with Randy, I could not think of a technical concern with the loop here except that things were working this way earlier. Having said that, since now the readers are also following the same state transitions, it became much simpler to introduce a loop without refactoring more than one function. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:1023: transaction->mode() == Transaction::NONE)) On 2017/04/06 at 17:37:13, jkarlin wrote: > I think you want: > > if ( !(transaction>mode() & Transaction::WRITE) || transaction->mode() == Transaction == NONE ) right? > > In which case, you can simplify that to: if (!transaction->mode() & Transaction::WRITE). Not sure of the feedback here. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:1027: return !entry->writer && entry->done_headers_queue.empty(); On 2017/04/06 at 17:37:13, jkarlin wrote: > I don't understand this conditional. Can you add a comment? This and the below conditions are actually doing more work than necessary. If the mode is write and it is a headers transaction then it has to be the one responsible for writing . Similarly if the mode is write and it is in done_headers_queue, it has to be the first in the queue and responsible for writing. Reduced them to: if (transaction == entry->headers_transaction || transaction == entry->done_headers_queue.front()) return true; https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:1041: auto i = std::find(entry->done_headers_queue.begin(), On 2017/04/06 at 17:37:13, jkarlin wrote: > s/i/it/ done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.cc... net/http/http_cache.cc:1104: auto it = entry->readers.find(transaction); On 2017/04/06 at 17:37:13, jkarlin wrote: > Can you add a TODO that this is a linear operation and it would be good to make it faster? readers is an unordered_set so average case is not linear but constant. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:268: // We implement a basic reader/writer lock for the disk cache entry. If On 2017/04/06 at 17:37:13, jkarlin wrote: > Pull this comment up above the definition of the ActiveEntry class. done and revised this comment for read-only transactions, since they are not special case anymore. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:279: // add_to_entry_queue-> headers_transaction -> done_headers_queue -> On 2017/04/06 at 17:37:13, jkarlin wrote: > I'd love to see the state machine description above the definition of ActiveEntry as well. done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:285: // add_to_entry_queue-> readers (once the data is written to the cache by On 2017/04/06 at 17:37:13, jkarlin wrote: > Why special case read-only transactions? It seems like it should follow the normal path of add_to_entry_queue-> headers_transaction-> done_headers_queue-> readers. > > After F2F we're in agreement about this. done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:393: // and the transaction will be notified about completion via its IO callback. On 2017/04/06 at 17:37:13, jkarlin wrote: > Comment on what error to expect on failure. done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:399: // to process queued transactions of the entry. Returns ERR_IO_PENDING. On 2017/04/06 at 17:37:13, jkarlin wrote: > Add "the transaction will be notified about completion via its IO callback" as in the comment for AddTransactionToEntry done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:413: // writing the response body. On 2017/04/06 at 17:37:13, jkarlin wrote: > Given the above method, and the name of this method, I'd expect the comment to say: > > "Returns true if this transaction can write response body bytes to the entry." > > Instead it's really a method that tries to answer two questions: > > 1) Can the transaction write to the response body? > 2) Did the transaction start writing to this entry but not yet finish? > > I believe we can split those into two functions: > > 1) CanTransactionWriteResponseBody > 2) IsTransactionWritingIncomplete > > WDYT? I tried doing that and made CanTransactionWriteResponseBody return true if transaction is the current writer and dcheck(CanTransactionWriteResponseBody) in DoCacheWriteData but one of the tests GET_Crazy416 fails because the transaction is still headers_transaction. It is a test which tests incorrect receiving of 416 for a non-range request. Removing this dcheck and this function and keeping one function IsTransactionWritingIncomplete to be called in the destructor. https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:446: void RemoveAllQueuedTransactions(ActiveEntry* entry, TransactionList* list); On 2017/04/06 at 17:37:13, jkarlin wrote: > Note that these methods need to appear in cc file in the same order. Right now ProcessEntryFailure appears above ProcessQueuedTransactions. done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:200: bool cancel_response = cache_->CanTransactionWriteResponseBody( On 2017/04/06 at 17:37:14, jkarlin wrote: > nit: s/cancel_response/writing_incomplete/ done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1577: On 2017/04/06 at 17:37:14, jkarlin wrote: > nit: remove newline done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1710: // cannot write to this entry. On 2017/04/06 at 17:37:14, jkarlin wrote: > Also mention that the transaction continues on without writing to the backend. done https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1841: return cache_->DoneWithResponseHeaders(entry_, this); On 2017/04/06 at 17:37:14, jkarlin wrote: > Since you're not passing a callback it's non-obvious that this is an async call. Can you add a comment that the cache will run the io_callback_? done
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...) win_chromium_compile_dbg_ng on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_comp...)
This patch removes the loop from ProcessDoneHeadersQueue and reverts back to post a task for every transaction. This is done because not only can a transaction's IO callback lead to the entry being destroyed (which can be fixed by setting the flag will_process_queued_transactions at the end, rather than start of OnProcessQueuedTransactions) but it can also lead to destroying the HttpCache itself (as seen in test EmbeddedTestServerThreadingTestInstantiation/EmbeddedTestServerThreadingTest.RunTest/0).
https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/500001/net/http/http_cache.h#... net/http/http_cache.h:413: // writing the response body. On 2017/04/07 21:11:16, shivanisha wrote: > On 2017/04/06 at 17:37:13, jkarlin wrote: > > Given the above method, and the name of this method, I'd expect the comment to > say: > > > > "Returns true if this transaction can write response body bytes to the entry." > > > > Instead it's really a method that tries to answer two questions: > > > > 1) Can the transaction write to the response body? > > 2) Did the transaction start writing to this entry but not yet finish? > > > > I believe we can split those into two functions: > > > > 1) CanTransactionWriteResponseBody > > 2) IsTransactionWritingIncomplete > > > > WDYT? > > > I tried doing that and made CanTransactionWriteResponseBody return true if > transaction is the current writer and dcheck(CanTransactionWriteResponseBody) in > DoCacheWriteData but one of the tests GET_Crazy416 fails because the transaction > is still headers_transaction. It is a test which tests incorrect receiving of > 416 for a non-range request. Removing this dcheck and this function and keeping > one function IsTransactionWritingIncomplete to be called in the destructor. Acknowledged. https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:982: void HttpCache::ProcessAddToEntryQueue(ActiveEntry* entry) { DCHECK that the queue isn't empty https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:995: can you remove the blank lines between line 990 and 996? https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1000: DCHECK(!entry->writer && !entry->done_headers_queue.empty()); Split into two DCHECKs. https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1006: if (transaction->mode() & Transaction::WRITE) { Can you DCHECK that we've never had a writer for this entry before? I worry that we might have two transactions that get into the headers queue with Transaction::WRITE mode. https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1009: // Response body is already written, convert to a reader. Can you add to the comment how we know that the response body is written? https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1032: if (response_code != 304) { DCHECK that there hasn't already been a writer for this ActiveEntry? https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1054: transaction->mode() == Transaction::NONE)) Needs braces since the condition is multi-line. https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1062: transaction == entry->done_headers_queue.front()) Needs braces since the condition is multi-line. https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1162: if (!entry->writer && !entry->done_headers_queue.empty()) { This one-liner has braces while the one below doesn't. At first glance this file doesn't use braces for one-liners so let's remove these braces. https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.h#... net/http/http_cache.h:260: add a // for line 260
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:982: void HttpCache::ProcessAddToEntryQueue(ActiveEntry* entry) { On 2017/04/11 at 14:38:28, jkarlin wrote: > DCHECK that the queue isn't empty done https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:995: On 2017/04/11 at 14:38:28, jkarlin wrote: > can you remove the blank lines between line 990 and 996? done https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1000: DCHECK(!entry->writer && !entry->done_headers_queue.empty()); On 2017/04/11 at 14:38:28, jkarlin wrote: > Split into two DCHECKs. done https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1006: if (transaction->mode() & Transaction::WRITE) { On 2017/04/11 at 14:38:28, jkarlin wrote: > Can you DCHECK that we've never had a writer for this entry before? I worry that > we might have two transactions that get into the headers queue with > Transaction::WRITE mode. Even if we have had a writer before, it's ok for this transaction to write. Consider this case: txn1 -> gets 200 -> writes response -> finishes txn2 -> gets 200 -> can still write response to this entry. The only invariant that should be true is that in done_headers_queue , only the first transaction is eligible to be a writer and none of the other transactions. Added this invariant in DoneWithResponseHeaders before adding a transaction to done_headers_queue. https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1009: // Response body is already written, convert to a reader. On 2017/04/11 at 14:38:28, jkarlin wrote: > Can you add to the comment how we know that the response body is written? done https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1032: if (response_code != 304) { On 2017/04/11 at 14:38:28, jkarlin wrote: > DCHECK that there hasn't already been a writer for this ActiveEntry? Not necessary. See comment above. https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1054: transaction->mode() == Transaction::NONE)) On 2017/04/11 at 14:38:28, jkarlin wrote: > Needs braces since the condition is multi-line. done https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1062: transaction == entry->done_headers_queue.front()) On 2017/04/11 at 14:38:28, jkarlin wrote: > Needs braces since the condition is multi-line. done https://codereview.chromium.org/2721933002/diff/580001/net/http/http_cache.cc... net/http/http_cache.cc:1162: if (!entry->writer && !entry->done_headers_queue.empty()) { On 2017/04/11 at 14:38:28, jkarlin wrote: > This one-liner has braces while the one below doesn't. At first glance this file doesn't use braces for one-liners so let's remove these braces. N/A now that this block will also have a return (in latest patch)
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: ios-simulator on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/bui...)
Still LGTM. https://codereview.chromium.org/2721933002/diff/620001/net/http/http_transact... File net/http/http_transaction_test_util.cc (right): https://codereview.chromium.org/2721933002/diff/620001/net/http/http_transact... net/http/http_transaction_test_util.cc:494: void MockNetworkTransaction::DeferNetworkStart(bool* defer) { nit, suggestion: This function seems trivial enough that I'd be inclined to say that tests that want this functionality can put it in the test file; I don't think it's worth adding cruft to the MockNetworkTransaction interface to have it centrally.
(There's a question in this message at the end regarding a failing bot test) The latest patch moves DeferNetworkStart to the test file as per Randy's feedback. It also fixes a bot failure (URLRequestTestHTTP.NetworkDelegateOnAuthRequiredSyncNoAction_GetFullRequestHeaders) which was coming due to the change that now DoAddToEntry always adds to the entry asynchronously. I have a question about suggestions to fix another bot failure SpdyNetworkTransactionTest.TestRawHeaderSizeSuccessfullRequest The issue is coming because URLRequest::OnHeadersComplete() is filling in the raw_header_size_ field by invoking GetTotalReceivedBytes() which is expected to go over all the layers to spdy stream and return the total received bytes expecting that should be the headers size. Its failing now because there is a task posted before a transaction gets added to the done_headers_queue and when that task is executed, it returns to the consumer and the spdy stream by that time has gone ahead and read more bytes and closed itself than just the headers and thus URLRequest fills raw_header_size_ with something more than just the headers bytes. I thought of a work around for this by doing something like below in URLRequest::OnHeadersComplete() HttpResponseInfo info; job_->GetResponseInfo(&info); raw_header_size_ = info.headers ? info.headers->raw_headers().size() : 0; but this doesn't work for Spdy because info.headers->raw_headers() returns the contents which have an additional status line added (https://cs.chromium.org/chromium/src/net/http/http_response_headers.cc?l=652) to the stream (not sure which consumers are dependent on that) and the spdy stream: status:200 hello:bye is getting stored as: HTTP/1.1 200 status:200 hello:bye I can think of adding a GetReceivedHeadersBytes() API to all the layers as another option provided Spdy and all streams are aware of header boundaries? That would be done in a separate CL and this CL will be dependent on that. Any suggestions on ways to unblock this CL? https://codereview.chromium.org/2721933002/diff/620001/net/http/http_transact... File net/http/http_transaction_test_util.cc (right): https://codereview.chromium.org/2721933002/diff/620001/net/http/http_transact... net/http/http_transaction_test_util.cc:494: void MockNetworkTransaction::DeferNetworkStart(bool* defer) { On 2017/04/11 at 19:56:45, Randy Smith (Not in Mondays) wrote: > nit, suggestion: This function seems trivial enough that I'd be inclined to say that tests that want this functionality can put it in the test file; I don't think it's worth adding cruft to the MockNetworkTransaction interface to have it centrally. done
> I have a question about suggestions to fix another bot failure SpdyNetworkTransactionTest.TestRawHeaderSizeSuccessfullRequest > > The issue is coming because URLRequest::OnHeadersComplete() is filling in the raw_header_size_ field by invoking GetTotalReceivedBytes() which is expected to go over all the layers to spdy stream and return the total received bytes expecting that should be the headers size. > > Its failing now because there is a task posted before a transaction gets added to the done_headers_queue and when that task is executed, it returns to the consumer and the spdy stream by that time has gone ahead and read more bytes and closed itself than just the headers and thus URLRequest fills raw_header_size_ with something more than just the headers bytes. > > I thought of a work around for this by doing something like below in URLRequest::OnHeadersComplete() > HttpResponseInfo info; > job_->GetResponseInfo(&info); > raw_header_size_ = info.headers ? info.headers->raw_headers().size() : 0; > > but this doesn't work for Spdy because info.headers->raw_headers() returns the contents which have an additional status line added (https://cs.chromium.org/chromium/src/net/http/http_response_headers.cc?l=652) to the stream (not sure which consumers are dependent on that) and the spdy stream: > status:200 > hello:bye > > is getting stored as: > > HTTP/1.1 200 > status:200 > hello:bye > > I can think of adding a GetReceivedHeadersBytes() API to all the layers as another option provided Spdy and all streams are aware of header boundaries? > > That would be done in a separate CL and this CL will be dependent on that. Any suggestions on ways to unblock this CL? Thought about this and it seems a simpler solution than having a new API would be to have a function in HttpResponseHeaders which can return the size of actual headers received , since this class is the one that added the status line, it will know when to return the length of the headers - length of the status line in a new function raw_headers_on_wire_size()
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
On 2017/04/12 at 18:12:53, shivanisha wrote: > > I have a question about suggestions to fix another bot failure SpdyNetworkTransactionTest.TestRawHeaderSizeSuccessfullRequest > > > > The issue is coming because URLRequest::OnHeadersComplete() is filling in the raw_header_size_ field by invoking GetTotalReceivedBytes() which is expected to go over all the layers to spdy stream and return the total received bytes expecting that should be the headers size. > > > > Its failing now because there is a task posted before a transaction gets added to the done_headers_queue and when that task is executed, it returns to the consumer and the spdy stream by that time has gone ahead and read more bytes and closed itself than just the headers and thus URLRequest fills raw_header_size_ with something more than just the headers bytes. > > > > I thought of a work around for this by doing something like below in URLRequest::OnHeadersComplete() > > HttpResponseInfo info; > > job_->GetResponseInfo(&info); > > raw_header_size_ = info.headers ? info.headers->raw_headers().size() : 0; > > > > but this doesn't work for Spdy because info.headers->raw_headers() returns the contents which have an additional status line added (https://cs.chromium.org/chromium/src/net/http/http_response_headers.cc?l=652) to the stream (not sure which consumers are dependent on that) and the spdy stream: > > status:200 > > hello:bye > > > > is getting stored as: > > > > HTTP/1.1 200 > > status:200 > > hello:bye > > > > I can think of adding a GetReceivedHeadersBytes() API to all the layers as another option provided Spdy and all streams are aware of header boundaries? > > > > That would be done in a separate CL and this CL will be dependent on that. Any suggestions on ways to unblock this CL? > > Thought about this and it seems a simpler solution than having a new API would be to have a function in HttpResponseHeaders which can return the size of actual headers received , since this class is the one that added the status line, it will know when to return the length of the headers - length of the status line in a new function raw_headers_on_wire_size() The status line is actually being added by the Spdy code itself and not the HttpResponseHeaders as I mentioned earlier. Tried the approach to save the raw-headers-on-wire-size in HttpResponseHeaders from Spdy and getting that size if it is set, passes that test but fails 14 other non spdy tests :( I am afraid the new API at all layers (or atleast in Job and HttpTransaction layers) might need to be added.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: ios-simulator on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/bui...)
> > Its failing now because there is a task posted before a transaction gets added > to the done_headers_queue and when that task is executed, it returns to the > consumer and the spdy stream by that time has gone ahead and read more bytes and > closed itself than just the headers and thus URLRequest fills raw_header_size_ > with something more than just the headers bytes. My first thought is let's reconsider how we can do without the extra posttask.
https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc... net/http/http_cache.cc:859: DCHECK(!(transaction->mode() & Transaction::WRITE)); DCHECK(entry->done_headers_queue.empty() || !(transaction->mode() & Transaction::WRITE)) https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc... net/http/http_cache.cc:940: void HttpCache::RemoveAllQueuedTransactions(ActiveEntry* entry, Can be in anonymous namespace https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc... net/http/http_cache.cc:971: void HttpCache::ProcessQueuedTransactions(ActiveEntry* entry) { This can be in the anonymous namespace. Doing so allows OnProcessQueuedTransactions, ProcessAddToEntryQueue, and ProcessDoneHeadersQueue to also be in the anonymous namespace. https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc... net/http/http_cache.cc:1066: // Write mode then it should be the future writer. Just checking the front of lowercase Write https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.h#... net/http/http_cache.h:256: // waiting transactions can start their headers phase in parallel. Headers Can you also describe what happens if the headers don't match? https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.h#... net/http/http_cache.h:397: // headers. The transaction is added to a queue and a task is posted Remove latter part of sentence, just need "when its response headers phase is complete." https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.h#... net/http/http_cache.h:398: // to process queued transactions of the entry. Returns ERR_IO_PENDING and the The detail of the transaction being added to a queue and a post task is implementation detail, not necessary in the comment.
https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc... net/http/http_cache.cc:971: void HttpCache::ProcessQueuedTransactions(ActiveEntry* entry) { On 2017/04/13 15:05:52, jkarlin wrote: > This can be in the anonymous namespace. Doing so allows > OnProcessQueuedTransactions, ProcessAddToEntryQueue, and ProcessDoneHeadersQueue > to also be in the anonymous namespace. Ah, to do that you'd need to pass a WeakPtr<HttpCache> around. Hmm, I guess nevermind :(
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
This patch fixes the failing spdy test by making DoneWithResponseHeaders return OK for the writer transaction synchronously instead of going through the done_headers_queue. As discussed with Josh f2f, this is also a good performance optimization since most entries will only have one transaction associated with it. Also addressed the latest feedback. https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc... net/http/http_cache.cc:859: DCHECK(!(transaction->mode() & Transaction::WRITE)); On 2017/04/13 at 15:05:52, jkarlin wrote: > DCHECK(entry->done_headers_queue.empty() || !(transaction->mode() & Transaction::WRITE)) done https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc... net/http/http_cache.cc:940: void HttpCache::RemoveAllQueuedTransactions(ActiveEntry* entry, On 2017/04/13 at 15:05:52, jkarlin wrote: > Can be in anonymous namespace ActiveEntry is private to HttpCache, so cannot move it to anonymous namespace. https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.cc... net/http/http_cache.cc:1066: // Write mode then it should be the future writer. Just checking the front of On 2017/04/13 at 15:05:52, jkarlin wrote: > lowercase Write done https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.h#... net/http/http_cache.h:256: // waiting transactions can start their headers phase in parallel. Headers On 2017/04/13 at 15:05:52, jkarlin wrote: > Can you also describe what happens if the headers don't match? done https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.h#... net/http/http_cache.h:397: // headers. The transaction is added to a queue and a task is posted On 2017/04/13 at 15:05:52, jkarlin wrote: > Remove latter part of sentence, just need "when its response headers phase is complete." done https://codereview.chromium.org/2721933002/diff/640001/net/http/http_cache.h#... net/http/http_cache.h:398: // to process queued transactions of the entry. Returns ERR_IO_PENDING and the On 2017/04/13 at 15:05:52, jkarlin wrote: > The detail of the transaction being added to a queue and a post task is implementation detail, not necessary in the comment. Removed and also revised the comment to mention that it will return OK for writer case.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: linux_chromium_tsan_rel_ng on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Patch 28 fixes the data races that were caught by tsan . The fix includes removes the code that accessed response body related fields (timestamp and size) in the disk_entry form a transaction's header phase functions. It also includes this invariant in a comment in the header file, since block file cache is not thread safe in accessing those fields. Patch 29 includes rebasing and test only changes to fix some new unit tests that were dependent on the ordering of net log events and this CL changes that ordering.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
On 2017/04/14 at 20:28:24, shivanisha wrote: > Patch 28 fixes the data races that were caught by tsan . The fix includes removes the code that accessed response body related fields (timestamp and size) in the disk_entry form a transaction's header phase functions. > It also includes this invariant in a comment in the header file, since block file cache is not thread safe in accessing those fields. > > Patch 29 includes rebasing and test only changes to fix some new unit tests that were dependent on the ordering of net log events and this CL changes that ordering. rdsmith@, jkarlin@, PTAL for the solution to the data race failures between cache worker thread and IO thread in block file cache. I was able to remove any accesses of the response body fields in the header states in HCT and vice versa. This will also be an invariant for this code going forward for which I have added a comment in the header file (until block file cache is fixed, atleast). Thanks!
My only concern is about the apparent low probability problem being opened up by not having the 2GB workaround in place if we're racing. https://codereview.chromium.org/2721933002/diff/660001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/660001/net/http/http_cache.cc... net/http/http_cache.cc:859: // e.g. while computing raw headers size.) crbug reference? https://codereview.chromium.org/2721933002/diff/660001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/660001/net/http/http_cache.h#... net/http/http_cache.h:400: // it becomes the writer and returns OK. In other cases the transaction it nit: Remove "it". https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1254: if (!cache_->IsWritingInProgress(entry_)) { This scares me. Does this mean that we are opening up the possibility for a rare race when we already have a writer and are dealing with a 2GB file & StopCaching() is called? I'd feel better just disabling the parallel validation for > 2GB files. The comment below suggests that this may be reasonable--the "do not release the cache entry--if another transaction wants to use the cache entry" would suggest that the writer is intending to hold on to the cache entry as long as it is writing. Can we extend that into your change and just (:-{) disable parallel validation in this case? https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.h:264: // accessing disk entry fields of the other index as that might lead to data I'm not sure what "of the other index" means in this context? https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.h:265: // races in some backends (like block file cache). I'd rephrase this with to talk about what the cache backend API contract does or does not guarantee, with a comment that the disk cache backend appears not to guarantee it and a pointer to the new bug. I'd also suggest that this comment probably belongs somewhere else; at least, I don't see the conceptual linkage to this area of the source. Maybe try to make the summary as pithy as possible (with a reference to the bug) and include it at both places IsWritingInProgress is used?
Nice! A few more comments. https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache.cc... net/http/http_cache.cc:1090: return entry->writer ? true : false; return entry->writer; https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1254: if (!cache_->IsWritingInProgress(entry_)) { Needs a comment so that we understand why we can't remove this condition in the future without careful consideration. https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.h:262: // Note that since headers and response body phases might be going on in s/might be going on in parallel/can occur in parallel/ https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.h:265: // races in some backends (like block file cache). I think what we really need is a DCHECK in the entry that verifies that there are no ongoing write operations when a synchronous method is called. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc... net/http/http_cache.cc:937: // If writer fails, all transactions restart the headers_transaction. Not sure what this comment means. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc... net/http/http_cache.cc:940: entry->headers_transaction = nullptr; It looks like the code is going to restart every transaction by calling io_callback with ERR_CACHE_RACE but the headers_transaction, which will get restarted when its io callback next runs. Can you add a comment making that explicit? https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc... net/http/http_cache.cc:1058: if (response_code != 304) { Seems like we don't need the response code, but instead a bool "is_match"? Instead, perhaps the caller should do the is_match check and only call this function if it'll need to write? https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc... net/http/http_cache.cc:1088: if (transaction == entry->headers_transaction || return transaction == entry->headers_transaction || transaction == entry->done_headers_queue.front(); https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.h#... net/http/http_cache.h:267: // synchronous in knowing that headers have been received) nit: Can you move the comments in the parenthesis here and put it in the implementation code instead? https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.h#... net/http/http_cache.h:459: bool IsWritingInProgress(ActiveEntry* entry) const; Please add a TODO to remove this function once we clean up the cache backend and link to the crbug you created for that. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:2966: // case. Can you add a crbug link to the comment? https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... File net/http/http_cache_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1405: EXPECT_EQ(4, cache.GetCountReaders(kSimpleGET_Transaction.url)); Perhaps: if (i > 0) { EXPECT_FALSE(cache.IsWriterPresent(kSimpleGET_Transaction.url)); EXPECT_EQ(kNumTransactions - i, cache.GetCountReaders(kSimpleGET_Transaction.url)) } https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1424: TEST(HttpCache, SimpleGET_ParallelValidationNoMatch) { Please make the changes in this test in all of the new tests. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1430: std::vector<Context*> context_list; std::vector<std::unique_ptr<Context>> context_list; https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1434: context_list.push_back(new Context()); push_back(base::MakeUnique<Context>()); https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1446: for (int i = 0; i < kNumTransactions; ++i) { for(auto context : context_list) { EXPECT_EQ(LOAD_STATE_WAITING_FOR_CACHE, context->trans->GetLoadState9)); } https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1461: EXPECT_EQ(3, cache.disk_cache()->create_count()); The comment above makes me think that we'd expect the create count to be 5 but it's 3? Can you add a comment to explain this behavior? https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1465: for (int i = 0; i < kNumTransactions; ++i) { for each loop as above https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1470: for (int i = 0; i < kNumTransactions; ++i) { for each loop https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1481: for (int i = 0; i < kNumTransactions; ++i) { For loop no longer needed with unique pointers above. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1866: Can you also add a test case where the writer is destroyed while a transaction is parallel validating headers? https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1997: // to process the headers in parallel with readers presnt on the entry. s/presnt/present/ https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:6669: // Since |pending| is currently vaidating the already written headers s/vaidating/validating/ https://codereview.chromium.org/2721933002/diff/700001/net/http/http_transact... File net/http/http_transaction_test_util.h (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_transact... net/http/http_transaction_test_util.h:283: CompletionCallback callback_; // used for pause and restart. Perhaps a more descriptive name? Such as deferred_start_callback_? https://codereview.chromium.org/2721933002/diff/700001/net/http/mock_http_cac... File net/http/mock_http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/mock_http_cac... net/http/mock_http_cache.cc:537: bool MockDiskCache::IsDiskEntryDoomed(const std::string& key) { It looks as if MockDiskCache::DoomEntry doesn't call entry->Doom() so it won't have is_doomed set? It won't be in entries_ either if DoomEntry was called. https://codereview.chromium.org/2721933002/diff/700001/net/http/mock_http_cac... net/http/mock_http_cache.cc:671: int MockHttpCache::GetCountReaders(const std::string& key) { return size_t https://codereview.chromium.org/2721933002/diff/700001/net/http/mock_http_cac... net/http/mock_http_cache.cc:678: int MockHttpCache::GetCountAddToEntryQueue(const std::string& key) { return size_t https://codereview.chromium.org/2721933002/diff/700001/net/http/mock_http_cac... net/http/mock_http_cache.cc:685: int MockHttpCache::GetCountDoneHeadersQueue(const std::string& key) { return size_t https://codereview.chromium.org/2721933002/diff/700001/net/http/mock_http_cac... File net/http/mock_http_cache.h (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/mock_http_cac... net/http/mock_http_cache.h:240: remove newline
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
Patchset #30 (id:720001) has been deleted
jkarlin@, rdsmith@, PTAL, thanks! This patch has all the latest feedback addressed. https://codereview.chromium.org/2721933002/diff/660001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/660001/net/http/http_cache.cc... net/http/http_cache.cc:859: // e.g. while computing raw headers size.) On 2017/04/19 at 17:26:34, Randy Smith (Not in Mondays) wrote: > crbug reference? done https://codereview.chromium.org/2721933002/diff/660001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/660001/net/http/http_cache.h#... net/http/http_cache.h:400: // it becomes the writer and returns OK. In other cases the transaction it On 2017/04/19 at 17:26:34, Randy Smith (Not in Mondays) wrote: > nit: Remove "it". done https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache.cc... net/http/http_cache.cc:1090: return entry->writer ? true : false; On 2017/04/20 at 15:05:29, jkarlin wrote: > return entry->writer; Gives compile time error in Windows https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1254: if (!cache_->IsWritingInProgress(entry_)) { On 2017/04/19 at 17:26:34, Randy Smith (Not in Mondays) wrote: > This scares me. Does this mean that we are opening up the possibility for a rare race when we already have a writer and are dealing with a 2GB file & StopCaching() is called? > > I'd feel better just disabling the parallel validation for > 2GB files. The comment below suggests that this may be reasonable--the "do not release the cache entry--if another transaction wants to use the cache entry" would suggest that the writer is intending to hold on to the cache entry as long as it is writing. Can we extend that into your change and just (:-{) disable parallel validation in this case? Talked with asanka@ yesterday about this and here is what I understand, happens: If a cache entry has a prefix of a >2G response already cached, then we start getting the rest of the response in 2G chunks via range requests and if StopCaching is invoked while any of these chunks are being read, then the state machine returns after the current chunk and does not send a range request for the next chunk. For this reason if the cached content-length header of the prefix imply a size > 2G then we pre-emptively send the request to the network. In this case if I also include dooming the entry here , then no further transaction will map to this entry and the race would not happen. Added doomentry here. Also added a comment here as per feedback. https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.h:262: // Note that since headers and response body phases might be going on in On 2017/04/20 at 15:05:29, jkarlin wrote: > s/might be going on in parallel/can occur in parallel/ N/A now that this comment is not needed here. https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.h:264: // accessing disk entry fields of the other index as that might lead to data On 2017/04/19 at 17:26:34, Randy Smith (Not in Mondays) wrote: > I'm not sure what "of the other index" means in this context? N/A. https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.h:265: // races in some backends (like block file cache). On 2017/04/20 at 15:05:29, jkarlin wrote: > I think what we really need is a DCHECK in the entry that verifies that there are no ongoing write operations when a synchronous method is called. Added new functions in ActiveEntry for data size, last used time and last modified time that asserts the transaction's state with respect to other transactions and used those functions in everywhere in the .cc file. Removed this comment from here and included the explanation in the .cc file along with the crbug number as to why we are checking if a writer is in progress. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc... net/http/http_cache.cc:937: // If writer fails, all transactions restart the headers_transaction. On 2017/04/20 at 15:05:29, jkarlin wrote: > Not sure what this comment means. Whoops, fixed the comment. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc... net/http/http_cache.cc:940: entry->headers_transaction = nullptr; On 2017/04/20 at 15:05:29, jkarlin wrote: > It looks like the code is going to restart every transaction by calling io_callback with ERR_CACHE_RACE but the headers_transaction, which will get restarted when its io callback next runs. Can you add a comment making that explicit? done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc... net/http/http_cache.cc:1058: if (response_code != 304) { On 2017/04/20 at 15:05:29, jkarlin wrote: > Seems like we don't need the response code, but instead a bool "is_match"? Instead, perhaps the caller should do the is_match check and only call this function if it'll need to write? Changed to use is_match boolean. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.cc... net/http/http_cache.cc:1088: if (transaction == entry->headers_transaction || On 2017/04/20 at 15:05:29, jkarlin wrote: > return transaction == entry->headers_transaction || transaction == entry->done_headers_queue.front(); done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.h#... net/http/http_cache.h:267: // synchronous in knowing that headers have been received) On 2017/04/20 at 15:05:30, jkarlin wrote: > nit: Can you move the comments in the parenthesis here and put it in the implementation code instead? Already part of the implementation, removed from here. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache.h#... net/http/http_cache.h:459: bool IsWritingInProgress(ActiveEntry* entry) const; On 2017/04/20 at 15:05:29, jkarlin wrote: > Please add a TODO to remove this function once we clean up the cache backend and link to the crbug you created for that. The todo is added in the code where this is invoked . Not adding it here as I can imagine it being used for any other purpose since it is a general method. https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:2966: // case. On 2017/04/20 at 15:05:31, jkarlin wrote: > Can you add a crbug link to the comment? done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... File net/http/http_cache_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1405: EXPECT_EQ(4, cache.GetCountReaders(kSimpleGET_Transaction.url)); On 2017/04/20 at 15:05:32, jkarlin wrote: > Perhaps: > > if (i > 0) { > EXPECT_FALSE(cache.IsWriterPresent(kSimpleGET_Transaction.url)); > EXPECT_EQ(kNumTransactions - i, cache.GetCountReaders(kSimpleGET_Transaction.url)) > } done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1424: TEST(HttpCache, SimpleGET_ParallelValidationNoMatch) { On 2017/04/20 at 15:05:38, jkarlin wrote: > Please make the changes in this test in all of the new tests. Done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1430: std::vector<Context*> context_list; On 2017/04/20 at 15:05:40, jkarlin wrote: > std::vector<std::unique_ptr<Context>> context_list; done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1434: context_list.push_back(new Context()); On 2017/04/20 at 15:05:33, jkarlin wrote: > push_back(base::MakeUnique<Context>()); done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1446: for (int i = 0; i < kNumTransactions; ++i) { On 2017/04/20 at 15:05:32, jkarlin wrote: > for(auto context : context_list) { > EXPECT_EQ(LOAD_STATE_WAITING_FOR_CACHE, context->trans->GetLoadState9)); > } It seems auto context doesn't work with unique_ptr but auto&& context does. Changed to auto&&. Using just auto gives compile time error:../../net/http/http_cache_unittest.cc:1373:12: error: call to deleted constructor of 'std::unique_ptr<net::(anonymous namespace)::Context, std::default_delete<net::(anonymous namespace)::Context> >' for(auto context : context_list) { ^ ~ ../../build/linux/debian_jessie_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:273:7: note: 'unique_ptr' has been explicitly marked deleted here unique_ptr(const unique_ptr&) = delete; https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1461: EXPECT_EQ(3, cache.disk_cache()->create_count()); On 2017/04/20 at 15:05:36, jkarlin wrote: > The comment above makes me think that we'd expect the create count to be 5 but it's 3? Can you add a comment to explain this behavior? done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1465: for (int i = 0; i < kNumTransactions; ++i) { On 2017/04/20 at 15:05:32, jkarlin wrote: > for each loop as above done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1470: for (int i = 0; i < kNumTransactions; ++i) { On 2017/04/20 at 15:05:35, jkarlin wrote: > for each loop done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1481: for (int i = 0; i < kNumTransactions; ++i) { On 2017/04/20 at 15:05:38, jkarlin wrote: > For loop no longer needed with unique pointers above. removed https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1866: On 2017/04/20 at 15:05:36, jkarlin wrote: > Can you also add a test case where the writer is destroyed while a transaction is parallel validating headers? That's tested in SimpleGET_ParallelValidationCancelWriter https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1997: // to process the headers in parallel with readers presnt on the entry. On 2017/04/20 at 15:05:38, jkarlin wrote: > s/presnt/present/ done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:6669: // Since |pending| is currently vaidating the already written headers On 2017/04/20 at 15:05:37, jkarlin wrote: > s/vaidating/validating/ done
https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache.cc... net/http/http_cache.cc:1090: return entry->writer ? true : false; On 2017/04/21 23:06:00, shivanisha wrote: > On 2017/04/20 at 15:05:29, jkarlin wrote: > > return entry->writer; > > Gives compile time error in Windows Then use 'return entry->writer != nullptr' https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... File net/http/http_cache_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1446: for (int i = 0; i < kNumTransactions; ++i) { On 2017/04/21 23:06:01, shivanisha wrote: > On 2017/04/20 at 15:05:32, jkarlin wrote: > > for(auto context : context_list) { > > EXPECT_EQ(LOAD_STATE_WAITING_FOR_CACHE, context->trans->GetLoadState9)); > > } > It seems auto context doesn't work with unique_ptr but auto&& context does. > Changed to auto&&. > > Using just auto gives compile time > error:../../net/http/http_cache_unittest.cc:1373:12: error: call to deleted > constructor of 'std::unique_ptr<net::(anonymous namespace)::Context, > std::default_delete<net::(anonymous namespace)::Context> >' > for(auto context : context_list) { > ^ ~ > ../../build/linux/debian_jessie_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:273:7: > note: 'unique_ptr' has been explicitly marked deleted here > unique_ptr(const unique_ptr&) = delete; Sorry, that was supposed to be auto& context. You don't want &&.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache.cc... net/http/http_cache.cc:1090: return entry->writer ? true : false; On 2017/04/24 at 15:57:26, jkarlin wrote: > On 2017/04/21 23:06:00, shivanisha wrote: > > On 2017/04/20 at 15:05:29, jkarlin wrote: > > > return entry->writer; > > > > Gives compile time error in Windows > > Then use 'return entry->writer != nullptr' done https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... File net/http/http_cache_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/700001/net/http/http_cache_un... net/http/http_cache_unittest.cc:1446: for (int i = 0; i < kNumTransactions; ++i) { On 2017/04/24 at 15:57:26, jkarlin wrote: > On 2017/04/21 23:06:01, shivanisha wrote: > > On 2017/04/20 at 15:05:32, jkarlin wrote: > > > for(auto context : context_list) { > > > EXPECT_EQ(LOAD_STATE_WAITING_FOR_CACHE, context->trans->GetLoadState9)); > > > } > > It seems auto context doesn't work with unique_ptr but auto&& context does. > > Changed to auto&&. > > > > Using just auto gives compile time > > error:../../net/http/http_cache_unittest.cc:1373:12: error: call to deleted > > constructor of 'std::unique_ptr<net::(anonymous namespace)::Context, > > std::default_delete<net::(anonymous namespace)::Context> >' > > for(auto context : context_list) { > > ^ ~ > > ../../build/linux/debian_jessie_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:273:7: > > note: 'unique_ptr' has been explicitly marked deleted here > > unique_ptr(const unique_ptr&) = delete; > > Sorry, that was supposed to be auto& context. You don't want &&. Done
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
One last thing and I think we're good. https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/680001/net/http/http_cache_tr... net/http/http_cache_transaction.h:265: // races in some backends (like block file cache). On 2017/04/21 23:06:00, shivanisha wrote: > On 2017/04/20 at 15:05:29, jkarlin wrote: > > I think what we really need is a DCHECK in the entry that verifies that there > are no ongoing write operations when a synchronous method is called. > > Added new functions in ActiveEntry for data size, last used time and last > modified time that asserts the transaction's state with respect to other > transactions and used those functions in everywhere in the .cc file. Removed > this comment from here and included the explanation in the .cc file along with > the crbug number as to why we are checking if a writer is in progress. Sorry, I meant meant putting the checks in the disk entry (e.g., EntryImpl). But that can be for another CL. https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache.cc... net/http/http_cache.cc:126: if (index == kResponseContentIndex) What about other indexes? The header comment says that it checks for all GetDataSize calls.
Again LGTM with the "Why is this safe for >2GB?" comment requested below. Thanks for your incredible patience! https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1208: // data race since cache thread can also access this. TODO(crbug.com/713354) Usual format is "TODO(<username> of person who can give more details): Fix this; see http://crbug.com/713354." (This has the same data, so just a suggestion.) https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1269: // http://crbug.com/89567. Include a comment as to why this is safe to have inside the conditional (i.e. that the doom below guarantees that the above race won't happen in the > 2GB case)?
asanka@chromium.org changed reviewers: + asanka@chromium.org
https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1208: // data race since cache thread can also access this. TODO(crbug.com/713354) On 2017/04/25 21:11:32, Randy Smith (Not in Mondays) wrote: > Usual format is "TODO(<username> of person who can give more details): Fix this; > see http://crbug.com/713354.%22 > > (This has the same data, so just a suggestion.) FYI: TODO(crbug.com/..) is now allowed in the style guide. But the convention is to place the TODO() before the description. Also https://groups.google.com/a/chromium.org/d/topic/chromium-dev/whBCkcY8xtA/dis...
On 2017/04/26 02:01:45, asanka wrote: > https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... > File net/http/http_cache_transaction.cc (right): > > https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... > net/http/http_cache_transaction.cc:1208: // data race since cache thread can > also access this. TODO(crbug.com/713354) > On 2017/04/25 21:11:32, Randy Smith (Not in Mondays) wrote: > > Usual format is "TODO(<username> of person who can give more details): Fix > this; > > see http://crbug.com/713354.%22 > > > > (This has the same data, so just a suggestion.) > > FYI: TODO(crbug.com/..) is now allowed in the style guide. But the convention is > to place the TODO() before the description. Also > https://groups.google.com/a/chromium.org/d/topic/chromium-dev/whBCkcY8xtA/dis... Ah, hadn't realized; thanks for the pointer! Please ignore my comment.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache.cc... net/http/http_cache.cc:126: if (index == kResponseContentIndex) On 2017/04/25 at 16:00:52, jkarlin wrote: > What about other indexes? The header comment says that it checks for all GetDataSize calls. As discussed f2f, removing the getter functions added here since Josh is working on CLs to fix this issue at the Block File Cache layer. https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1269: // http://crbug.com/89567. On 2017/04/25 at 21:11:32, Randy Smith (Not in Mondays) wrote: > Include a comment as to why this is safe to have inside the conditional (i.e. that the doom below guarantees that the above race won't happen in the > 2GB case)? done
Thank you Randy for a detailed and thoughtful review!
lgtm! with nits https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1208: // data race since cache thread can also access this. TODO(crbug.com/713354) On 2017/04/26 02:01:45, asanka wrote: > On 2017/04/25 21:11:32, Randy Smith (Not in Mondays) wrote: > > Usual format is "TODO(<username> of person who can give more details): Fix > this; > > see http://crbug.com/713354.%22 > > > > (This has the same data, so just a suggestion.) > > FYI: TODO(crbug.com/..) is now allowed in the style guide. But the convention is > to place the TODO() before the description. Also > https://groups.google.com/a/chromium.org/d/topic/chromium-dev/whBCkcY8xtA/dis... > This still needs to be addressed. https://codereview.chromium.org/2721933002/diff/780001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/780001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1257: // could lead to a race TODO(crbug.com/713354)). nit: This is hard to read, it sounds a big like Yoda :) Could you rewrite the above? Also, the TODO(crbug.com/713354) should start a sentence, not end it.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/760001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1208: // data race since cache thread can also access this. TODO(crbug.com/713354) On 2017/04/26 at 17:11:01, jkarlin wrote: > On 2017/04/26 02:01:45, asanka wrote: > > On 2017/04/25 21:11:32, Randy Smith (Not in Mondays) wrote: > > > Usual format is "TODO(<username> of person who can give more details): Fix > > this; > > > see http://crbug.com/713354.%22 > > > > > > (This has the same data, so just a suggestion.) > > > > FYI: TODO(crbug.com/..) is now allowed in the style guide. But the convention is > > to place the TODO() before the description. Also > > https://groups.google.com/a/chromium.org/d/topic/chromium-dev/whBCkcY8xtA/dis... > > > > This still needs to be addressed. done
Thanks Josh for such a detailed and thoughtful review! https://codereview.chromium.org/2721933002/diff/780001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/780001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1257: // could lead to a race TODO(crbug.com/713354)). On 2017/04/26 at 17:11:01, jkarlin wrote: > nit: This is hard to read, it sounds a big like Yoda :) Could you rewrite the above? > > Also, the TODO(crbug.com/713354) should start a sentence, not end it. Simplified
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
The CQ bit was checked by shivanisha@chromium.org
The patchset sent to the CQ was uploaded after l-g-t-m from rdsmith@chromium.org, jkarlin@chromium.org Link to the patchset: https://codereview.chromium.org/2721933002/#ps800001 (title: "Comments simplified")
CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
CQ is committing da patch. Bot data: {"patchset_id": 800001, "attempt_start_ts": 1493236873205250, "parent_rev": "04951aeb4b00ccc6454d6b757fc7b537e83d4a90", "commit_rev": "1e2e347f957ef889aaee527bb757849f76e8a808"}
Message was sent while issue was closed.
Description was changed from ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. BUG=472740 ========== to ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. BUG=472740 Review-Url: https://codereview.chromium.org/2721933002 Cr-Commit-Position: refs/heads/master@{#467426} Committed: https://chromium.googlesource.com/chromium/src/+/1e2e347f957ef889aaee527bb757... ==========
Message was sent while issue was closed.
Committed patchset #33 (id:800001) as https://chromium.googlesource.com/chromium/src/+/1e2e347f957ef889aaee527bb757...
Message was sent while issue was closed.
Congratulations!
Message was sent while issue was closed.
On 2017/04/26 20:39:13, Randy Smith (Not in Mondays) wrote: > Congratulations! Indeed. There are 170 messages and 394 in-line comments over 33 patchsets. This is some marathon.
Message was sent while issue was closed.
A revert of this CL (patchset #33 id:800001) has been created in https://codereview.chromium.org/2847653002/ by warx@chromium.org. The reason for reverting is: Breaks tricky-tot-chrome-pfq-informational audio_CrasSanity autotest: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/tricky-tot-ch....
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by shivanisha@chromium.org
Patches 34, 35 and 36 fix some of the crashes that came. I am looking into more and will upload more fixes: - Patch 34 fixes crbug.com/715913: The DCHECK was getting hit because a HEAD request HttpCache::Transaction for a cached entry was not following the same state in the ActiveEntry as a GET request i.e headers_transaction->done_headers_queue->reader. This was initially done because a HEAD request should not conceptually become a reader, because it does not read any body. What happens in practice is a little different though. Here is what happens: URLRequestHttpJob does not distinguish between a HEAD and a GET request and sends a request for response body either ways. If its a fresh request from the network, HttpStreamParser state machine ensures that the Read does not return anything even if server sends a body (https://cs.chromium.org/chromium/src/net/http/http_stream_parser.cc?q=http_st...). If its a cached response, HttpCache::Transaction ensures that nothing is read from cached response body (https://cs.chromium.org/chromium/src/net/http/http_cache_transaction.cc?dr=C&...) Unfortunately, there did not exist any unit tests in URLRequestHttpJob to test a HEAD request. Thus fixing this issue by making HEAD requests also follow the same states in the ActiveEntry as a GET request. Also adding 3 unit tests that test the HEAD request at the URLRequestHttpJob layer: fresh HEAD request with no content, fresh HEAD request and server also sends content(server error), cached HEAD request. - Patch 35 fixes the telemetry_perf_unittest crash coming in patch 34. It removes the invariants of modes for partial requests so that they only apply to non-range requests. - Patch 36 fixes the asan failure in crbug.com/715911. Also tried that bot with the CL and its running fine now. The issue was that HttpCache::MetadataWriter declares the |request_info_| member after the |transaction_| member. Since |transaction_|'s destructor accessed |request_info_|, it was accessing a freed memory. Fixing it by changing the order in MetadataWriter but also using a cached value of method in HttpCache::Transaction instead of request_->method everytime it is needed. I am not sure why these don't show up in the commit queue bot: linux_chromium_asan_rel_ng
Patch 37 fixes HttpTransaction::Start()'s comment for |*request_info| for range requests and fixes usage of request_ in RecordHistograms as it should not be accessed at that point since that could be after headers have been received.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
On 2017/05/01 at 15:37:39, shivanisha wrote: > Patch 37 fixes HttpTransaction::Start()'s comment for |*request_info| for range requests and fixes usage of request_ in RecordHistograms as it should not be accessed at that point since that could be after headers have been received. Patch 35: Fixed partial transactions handling in ActiveEntry also fixes the clusterfuzz reported issue: Here are the details: Tested the clusterfuzz test reported in crbug.com/715974 (https://clusterfuzz.com/v2/testcase-detail/5171679371460608?noredirect=1) and consistently the crash is coming locally on the reverted CL with the URL: https://aeginaportal.gr/ command: ./out/asan/chrome --no-sandbox | tools/valgrind/asan/asan_symbolize.py gn args: enable_ipc_fuzzer = true ffmpeg_branding = "Chrome" is_component_build = false proprietary_codecs = true sanitizer_coverage_flags = "edge" v8_enable_verify_heap = true is_asan = true is_lsan = true enable_nacl = false is_debug = false The same command , gn args and the same URL do not crash on the latest patch. The crash reason was that DoneReadingFromEntry was being invoked for a transaction incorrectly. It should have been found in done_headers_queue/headers_transaction/writer but it incorrectly did not, thus assuming it should be a reader. Patch 35: Fixed partial transactions handling in ActiveEntry fixed this issue. Before the fix the state transition in DoneWithResponseHeaders looked like this: if (transaction->mode() & Transaction::WRITE) { DCHECK(entry->done_headers_queue.empty()); DCHECK(!entry->writer); entry->writer = transaction; Since dchecks were not getting hit in the asan build that clusterfizz was using , instead of failing at DCHECK(!entry->writer), it was actually over-writing writer transaction with another transaction and thus losing the pointer to the original writer. Scary!
Reviewed PS34-37; haven't reviewed after PS37 as there haven't been posts requesting review for those. One high level comment on PS#36: I'm confused as to why there are two changes in this PS (caching the method *and* moving the transaction to the end of MetadataWriter so that it's destroyed before the request_info_). It seems like either of these would have solved the problem described--why do both? Is there some other problem being solved I'm missing? https://codereview.chromium.org/2721933002/diff/800001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/800001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1851: return OK; At what point is the HC::T removed from the headers transaction in the entry? I suspect it's when the transaction is destroyed, which I'm concerned is when the request is destroyed, which I could imagine being a long time after this point. Would a request that returns successfully and isn't destroyed block further activity on this entry? Is it reasonable to have a test for that? https://codereview.chromium.org/2721933002/diff/840001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/840001/net/http/http_cache.cc... net/http/http_cache.cc:1059: return false; I'm trying to wrap my head around this change, and so far not succeeding. I believe this was done in PS35, but I'm not seeing its relationship to the description of what PS35 was for. Could you give me a bit more context for this specific code change? (I have a vague sense of this possibly being related to partial transactions being able to have WRITE mode set even if they're readers, as they may be writers for one range and readers for another. But why does that disallow there being a writer ahead of them in the state machine when they'll eventually be a writer?)
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
PTAL, thanks! Patch 39 fixes the re-setting of cache_entry_status_ which was failing webkit tests in linux_trusty_blink_dbg bot. The tests are successful now. Patch 40 makes IsTransactionWritingIncomplete clearer for range requests by adding comments and re-ordering conditions. https://codereview.chromium.org/2721933002/diff/800001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/800001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:1851: return OK; On 2017/05/03 at 16:16:00, Randy Smith (Not in Mondays) wrote: > At what point is the HC::T removed from the headers transaction in the entry? I suspect it's when the transaction is destroyed, which I'm concerned is when the request is destroyed, which I could imagine being a long time after this point. Would a request that returns successfully and isn't destroyed block further activity on this entry? Is it reasonable to have a test for that? Is the question specifically on HEAD transactions? If yes, then patch 34 actually removes this if and the transaction will move to done_headers_queue. https://codereview.chromium.org/2721933002/diff/840001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/840001/net/http/http_cache.cc... net/http/http_cache.cc:1059: return false; On 2017/05/03 at 16:16:00, Randy Smith (Not in Mondays) wrote: > I'm trying to wrap my head around this change, and so far not succeeding. I believe this was done in PS35, but I'm not seeing its relationship to the description of what PS35 was for. Could you give me a bit more context for this specific code change? > > (I have a vague sense of this possibly being related to partial transactions being able to have WRITE mode set even if they're readers, as they may be writers for one range and readers for another. But why does that disallow there being a writer ahead of them in the state machine when they'll eventually be a writer?) This condition is basically saying that: if there is another transaction that is the writer then do not consider this the future writer for this range. For non-range requests, its ok to infer this information from the conditions ahead involving the mode since they will always have !WRITE mode if there is another writer but not for range requests. Putting some comments in the code and re-ordering the conditions to clarify this. PTAL.
Patch 41 simplifies the lifetime requirement of request_info while sending it in HttpTransaction::Start(). Earlier the requirement was that the request_info should not be used by the transaction after final response headers are received. That is good for all transactions except HttpCache::Transaction because if its using range requests it would go to headers phase repeatedly for multiple ranges. Thus it is simpler to tell the consumer that the request info should be alive till the consumer is alive and in case the transaction outlives the consumer it should not access request info. This was the initial motivation to have this comment in the first place, anyways.
On 2017/05/03 at 20:04:33, shivanisha wrote: > Patch 41 simplifies the lifetime requirement of request_info while sending it in HttpTransaction::Start(). Earlier the requirement was that the request_info should not be used by the transaction after final response headers are received. That is good for all transactions except HttpCache::Transaction because if its using range requests it would go to headers phase repeatedly for multiple ranges. > > Thus it is simpler to tell the consumer that the request info should be alive till the consumer is alive and in case the transaction outlives the consumer it should not access request info. This was the initial motivation to have this comment in the first place, anyways. In light of the above lifetime requirement of HttpRequestInfo, answering Randy's high level comment on patch 36: The specific bug would have been fixed by either of the two approaches but doing both imply that: - MetadataWriter adheres to the API, and - HC::T is robust to another future consumer that might reverse the ordering. Also, in itself I feel since we are accessing request_->method so many times, its good to save it independently. (Its in line with HttpNetworkTransaction/Streams that do similar things since they may outlive the consumer that owns request info.)
Description was changed from ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. BUG=472740 Review-Url: https://codereview.chromium.org/2721933002 Cr-Commit-Position: refs/heads/master@{#467426} Committed: https://chromium.googlesource.com/chromium/src/+/1e2e347f957ef889aaee527bb757... ========== to ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. BUG=472740, 715913, 715974, 715920, 715911, Review-Url: https://codereview.chromium.org/2721933002 Cr-Commit-Position: refs/heads/master@{#467426} Committed: https://chromium.googlesource.com/chromium/src/+/1e2e347f957ef889aaee527bb757... ==========
Working on adding a few new unit tests, specifically for range requests. Will upload a patch once those are done.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
rdsmith@, PTAL, thanks! The latest patch includes unit tests for range requests and some refactoring including the control flow : ~Transaction -> DoneWithEntry for processing the cancel case to make things clearer for partial requests. Feel free to let me know if we need a quick VC to go over any of the new patches. The bugs (linked in this CL) raised after the original commit should be fixed now. The crashes that came in canary came in different points in the code because the DCHECKS were not enabled. The URLs that were reported in those crashes (amazon.in and gmail) started working fine after the patches that fix HEAD and range requests. One issue that was raised in Chromium OS sanity checks is not a crash but a test failure and I will work with them to know the next steps there.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
rdsmith@, PTAL, thanks! This patch converts the invariants of the Active Entry state transitions in DoneWithResponseHeaders from DCHECK to CHECK. A lot of crashes were reported in canary but because these were DCHECKs , it crashed at different points in the code.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
On 2017/05/08 at 19:33:42, shivanisha wrote: > rdsmith@, PTAL, thanks! > > This patch converts the invariants of the Active Entry state transitions in DoneWithResponseHeaders from DCHECK to CHECK. A lot of crashes were reported in canary but because these were DCHECKs , it crashed at different points in the code. rdsmith@, ping?
If you're going to change the DCHECKs to CHECKs, can you add a TODO() to change them back with a bug link? (I believe you're adding them to figure out why things are crashing in canary, and that you're intending to remove them after those issues are resolved--let me know if I have that wrong.) https://codereview.chromium.org/2721933002/diff/840001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/840001/net/http/http_cache.cc... net/http/http_cache.cc:1059: return false; On 2017/05/03 19:50:18, shivanisha wrote: > On 2017/05/03 at 16:16:00, Randy Smith (Not in Mondays) wrote: > > I'm trying to wrap my head around this change, and so far not succeeding. I > believe this was done in PS35, but I'm not seeing its relationship to the > description of what PS35 was for. Could you give me a bit more context for this > specific code change? > > > > (I have a vague sense of this possibly being related to partial transactions > being able to have WRITE mode set even if they're readers, as they may be > writers for one range and readers for another. But why does that disallow there > being a writer ahead of them in the state machine when they'll eventually be a > writer?) > > This condition is basically saying that: if there is another transaction that is > the writer then do not consider this the future writer for this range. > For non-range requests, its ok to infer this information from the conditions > ahead involving the mode since they will always have !WRITE mode if there is > another writer but not for range requests. Putting some comments in the code and > re-ordering the conditions to clarify this. PTAL. Acknowledged. https://codereview.chromium.org/2721933002/diff/920001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/920001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:2135: restart_info_.cache_entry_status = cache_entry_status_; So it looks like the only functional change in this patch is this one: preserving the cache_entry_status_ across restarts if-and-only-if it was set to ENTRY_OTHER because of a range request. That seems like weird corner case behavior that's likely to lead to bugs and confusion in the future--naively, we should always be saving the cache_entry_status_ on a restart, or at a minimum always saving it for ENTRY_OTHER. Can you include some comments (in the header file at the data structure member, I think?) as to why this value is only valid in this one corner case? https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... File net/http/http_transaction.h (right): https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... net/http/http_transaction.h:55: // The consumer should ensure that request_info points to a valid value till nit: "till" -> "while" https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... net/http/http_transaction.h:56: // the consumer is alive; after that point, the HttpTransaction will not How does the HttpTransaction know when its consumer dies? (I had thought we kept the API contract to being "the HttpTransaction will be destroyed when its consumer is destroyed"; i.e. the HttpTransaction would be owned by its consumer. In that case, I think this requirement translates to "The consumer must ensure that |*request_info| remains alive for the lifetime of the HttpTransaction.", which is both a simple and more common interface contract.) https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.c... net/http/http_cache.cc:847: CHECK(is_partial || (!entry->writer && entry->done_headers_queue.empty())); This is a (check-level) behavior change. Crashes used to happen if |is_partial && !entry->done_headers_queue.empty()| and no longer will. The description of this PS didn't mention a change in check behavior, just changing from DCHECK->CHECK. Why the change?
Randy, addressed your feedback, PTAL, Thanks. https://codereview.chromium.org/2721933002/diff/920001/net/http/http_cache_tr... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/920001/net/http/http_cache_tr... net/http/http_cache_transaction.cc:2135: restart_info_.cache_entry_status = cache_entry_status_; On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > So it looks like the only functional change in this patch is this one: preserving the cache_entry_status_ across restarts if-and-only-if it was set to ENTRY_OTHER because of a range request. That seems like weird corner case behavior that's likely to lead to bugs and confusion in the future--naively, we should always be saving the cache_entry_status_ on a restart, or at a minimum always saving it for ENTRY_OTHER. Can you include some comments (in the header file at the data structure member, I think?) as to why this value is only valid in this one corner case? The idea was to set restart_info_.cache_entry_status whenever it gets set in HttpCache::Transaction::SetRequest() as SetRequest will not be invoked again when the request is restarted. Putting it in the end of the function to make it unconditional. Thanks for pointing this out. https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... File net/http/http_transaction.h (right): https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... net/http/http_transaction.h:55: // The consumer should ensure that request_info points to a valid value till On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > nit: "till" -> "while" done https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... net/http/http_transaction.h:56: // the consumer is alive; after that point, the HttpTransaction will not On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > How does the HttpTransaction know when its consumer dies? > > (I had thought we kept the API contract to being "the HttpTransaction will be destroyed when its consumer is destroyed"; i.e. the HttpTransaction would be owned by its consumer. In that case, I think this requirement translates to "The consumer must ensure that |*request_info| remains alive for the lifetime of the HttpTransaction.", which is both a simple and more common interface contract.) That's true for HttpCache::Transaction but not for HttpNetworkTransaction since when we land parallel writing, it will be owning HttpNetworkTransaction even after consumer has died. https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.c... net/http/http_cache.cc:847: CHECK(is_partial || (!entry->writer && entry->done_headers_queue.empty())); On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > This is a (check-level) behavior change. Crashes used to happen if |is_partial && !entry->done_headers_queue.empty()| and no longer will. The description of this PS didn't mention a change in check behavior, just changing from DCHECK->CHECK. Why the change? Ah right. I forgot to mention that in the patch description. The patch that changed the behavior of the dcheck to include partial cases was originally patch 35, but it was incomplete because it made !entry->writer true only for non-partial requests but not entry->done_headers_queue.empty(). Basically, in case of partial , since any request can have write mode bit set irrespective of whether there is a writer or not or whether there is a done_headers_queue transaction or not. Responding to the high level question here as well, my intention here was to make these assertions as CHECKs permanently since these are important invariants of the code and if they ever break in production , they should be easy to see from the backtrace. I did find a few more CHECK assertions in net/. Let me know if that's a concern.
Patchset #44 (id:1020001) has been deleted
Asymptotically approaching agreement :-}. (And I suspect the problem with the interface contract is that I'm not understanding what you're trying to say rather than that it's wrong, but that should probably drive a clearer comment, so I'm going to keep poking at it.) https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... File net/http/http_transaction.h (right): https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... net/http/http_transaction.h:56: // the consumer is alive; after that point, the HttpTransaction will not On 2017/05/10 14:46:03, shivanisha wrote: > On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > > How does the HttpTransaction know when its consumer dies? > > > > (I had thought we kept the API contract to being "the HttpTransaction will be > destroyed when its consumer is destroyed"; i.e. the HttpTransaction would be > owned by its consumer. > In that case, I think this requirement translates to "The consumer must ensure > that |*request_info| remains alive for the lifetime of the HttpTransaction.", > which is both a simple and more common interface contract.) > > That's true for HttpCache::Transaction but not for HttpNetworkTransaction since > when we land parallel writing, it will be owning HttpNetworkTransaction even > after consumer has died. > Ok, I think we're using terminology differently: When I say "consumer" for a particular network transaction, I mean the immediate class that's driving it. So the consumer for the HttpNetworkTransaction would be the owning HttpCacheTransaction. If the lifetime for |*request_info| is the lifetime of the owning URLRequestHttpJob, then my concern is even stronger. How is the HttpNetworkTransaction going to know when that dies? If it can't, how can this interface contract be useful for the HttpNetworkTransaction if it can't know? I'm inclined to think that interface contracts with regard to time should be written s.t. both the (immediate) consumer and implementing class have access to the event occurrences that are mentioned. Otherwise it doesn't really make sense as an interface contract. But I suspect there's something I'm missing about event visibility here. https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.c... net/http/http_cache.cc:847: CHECK(is_partial || (!entry->writer && entry->done_headers_queue.empty())); On 2017/05/10 14:46:03, shivanisha wrote: > On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > > This is a (check-level) behavior change. Crashes used to happen if > |is_partial && !entry->done_headers_queue.empty()| and no longer will. The > description of this PS didn't mention a change in check behavior, just changing > from DCHECK->CHECK. Why the change? > > Ah right. I forgot to mention that in the patch description. The patch that > changed the behavior of the dcheck to include partial cases was originally patch > 35, but it was incomplete because it made !entry->writer true only for > non-partial requests but not entry->done_headers_queue.empty(). Basically, in > case of partial , since any request can have write mode bit set irrespective of > whether there is a writer or not or whether there is a done_headers_queue > transaction or not. Gotcha; that makes sense. > Responding to the high level question here as well, my intention here was to > make these assertions as CHECKs permanently since these are important invariants > of the code and if they ever break in production , they should be easy to see > from the backtrace. I did find a few more CHECK assertions in net/. Let me know > if that's a concern. Hmmm. I don't think of that as the normal idiom. I think of CHECKs being in production code in two cases: a) We're doing a run against canary to figure out where a particular failure is really happening, and b) the consequences of the CHECK not being there and the code continuing to execute are clearly so bad that it's better for the end user that the browser crash than that we blow through the CHECK. I don't think either applies here. I don't claim to be a complete authority on this issue--if you wanted to run it by Matt, I'd happily accept his override. But my call would be that these CHECKs shouldn't be in on a long-term basis.
For the canary (60.0.3082.0) crashes, all but one of them can be speculated to be fixed because of the fixes in patches 34 (Cached HEAD request handling fixed) and 35 (Fixed partial transactions handling in ActiveEntry). Most of them are related to the inconsistency where a transaction thinks it is associated with the entry while the entry does not have a reference to it (this happened for partial requests) OR transaction assuming that it is a reader in the entry while it was the headers_transaction (this happened for HEAD requests). For the one report, that I am not able to put a root cause to, is in the shutdown case where HttpCache is trying to access a destroyed entry? Josh/Randy, Does something come in your mind regarding how this CL could be leading to this crash: example report ids are 5c8dbcee80000000 or 34c6757b30000000 or 6f014fd0d0000000 ? Thanks. If not, may be I'll wait to see if it gets fixed with the fixes now done in the canary when this lands. All of these crashes are at statement number 353 in http_cache.cc (see patch 33) entry->readers.clear();
On 2017/05/11 17:50:45, shivanisha wrote: > For the canary (60.0.3082.0) crashes, all but one of them can be speculated to > be fixed because of the fixes in patches 34 (Cached HEAD request handling fixed) > and 35 (Fixed partial transactions handling in ActiveEntry). Most of them are > related to the inconsistency where a transaction thinks it is associated with > the entry while the entry does not have a reference to it (this happened for > partial requests) OR transaction assuming that it is a reader in the entry while > it was the headers_transaction (this happened for HEAD requests). > > For the one report, that I am not able to put a root cause to, is in the > shutdown case where HttpCache is trying to access a destroyed entry? > > Josh/Randy, Does something come in your mind regarding how this CL could be > leading to this crash: example report ids are 5c8dbcee80000000 or > 34c6757b30000000 or 6f014fd0d0000000 ? Thanks. > If not, may be I'll wait to see if it gets fixed with the fixes now done in the > canary when this lands. > > All of these crashes are at statement number 353 in http_cache.cc (see patch 33) > entry->readers.clear(); Just to make sure I understand the details: What makes you say that it's a destroyed entry? Just that there's a problem on clearing a std:: container of pointers? (FWIW, I can't think of any other reason why that would crash other than use-after-free, but if you have other information confirming that, it's probably good that I understand it.)
On 2017/05/11 18:38:02, Randy Smith (Not in Mondays) wrote: > On 2017/05/11 17:50:45, shivanisha wrote: > > For the canary (60.0.3082.0) crashes, all but one of them can be speculated to > > be fixed because of the fixes in patches 34 (Cached HEAD request handling > fixed) > > and 35 (Fixed partial transactions handling in ActiveEntry). Most of them are > > related to the inconsistency where a transaction thinks it is associated with > > the entry while the entry does not have a reference to it (this happened for > > partial requests) OR transaction assuming that it is a reader in the entry > while > > it was the headers_transaction (this happened for HEAD requests). > > > > For the one report, that I am not able to put a root cause to, is in the > > shutdown case where HttpCache is trying to access a destroyed entry? > > > > Josh/Randy, Does something come in your mind regarding how this CL could be > > leading to this crash: example report ids are 5c8dbcee80000000 or > > 34c6757b30000000 or 6f014fd0d0000000 ? Thanks. > > If not, may be I'll wait to see if it gets fixed with the fixes now done in > the > > canary when this lands. > > > > All of these crashes are at statement number 353 in http_cache.cc (see patch > 33) > > entry->readers.clear(); > > Just to make sure I understand the details: What makes you say that it's a > destroyed entry? Just that there's a problem on clearing a std:: container of > pointers? (FWIW, I can't think of any other reason why that would crash other > than use-after-free, but if you have other information confirming that, it's > probably good that I understand it.) The only idea I come up with from examining the code is a failure of the DCHECK inside of DeactivateEntry that confirms the entry being erased is the same as the one passed in. It might be worthwhile elevating that one to a CHECK and seeing if it trips. Having said that, I don't think that would cause this problem--that would leave the active entry in the list to be grabbed again, but it wouldn't have been deleted, just nulled out. The deletion is tied to the erasure from the list (because the list holds the entry with a unique_ptr<>) so in theory it should always be undeleted while its on the list and deleted after it's off. Which means this is weird and "can't happen". (Though it might be a wandering pointer bug, which would be all sorts of fun to track down.) A really large hammer: Create a data structure on HttpCache that is used to store a large set of stack traces from deletion of an HttpCache::ActiveEntry. Store the last 100 or something like that. Have that DS be looked up by pointer. Erase elements if a new ActiveEntry is created with the same pointer. Do an explicit crash if there's a second deletion of an active entry that's already present in the set, after getting the stack trace from the first deletion onto the stack. You'll then need to look at that stack in the minidump with a debugger, but if the problem is indeed a free-after-free, that would get you a long step towards figuring out what's going on. I'll keep musing on this a bit and see if I can come up with anything else to suggest. FWIW, I don't like the pattern used in ~HttpCache around the active entries, or
On 2017/05/11 18:51:07, Randy Smith (Not in Mondays) wrote: > On 2017/05/11 18:38:02, Randy Smith (Not in Mondays) wrote: > > On 2017/05/11 17:50:45, shivanisha wrote: > > > For the canary (60.0.3082.0) crashes, all but one of them can be speculated > to > > > be fixed because of the fixes in patches 34 (Cached HEAD request handling > > fixed) > > > and 35 (Fixed partial transactions handling in ActiveEntry). Most of them > are > > > related to the inconsistency where a transaction thinks it is associated > with > > > the entry while the entry does not have a reference to it (this happened for > > > partial requests) OR transaction assuming that it is a reader in the entry > > while > > > it was the headers_transaction (this happened for HEAD requests). > > > > > > For the one report, that I am not able to put a root cause to, is in the > > > shutdown case where HttpCache is trying to access a destroyed entry? > > > > > > Josh/Randy, Does something come in your mind regarding how this CL could be > > > leading to this crash: example report ids are 5c8dbcee80000000 or > > > 34c6757b30000000 or 6f014fd0d0000000 ? Thanks. > > > If not, may be I'll wait to see if it gets fixed with the fixes now done in > > the > > > canary when this lands. > > > > > > All of these crashes are at statement number 353 in http_cache.cc (see patch > > 33) > > > entry->readers.clear(); > > > > Just to make sure I understand the details: What makes you say that it's a > > destroyed entry? Just that there's a problem on clearing a std:: container of > > pointers? (FWIW, I can't think of any other reason why that would crash other > > than use-after-free, but if you have other information confirming that, it's > > probably good that I understand it.) > > The only idea I come up with from examining the code is a failure of the DCHECK > inside of DeactivateEntry that confirms the entry being erased is the same as > the one passed in. It might be worthwhile elevating that one to a CHECK and > seeing if it trips. Having said that, I don't think that would cause this > problem--that would leave the active entry in the list to be grabbed again, but > it wouldn't have been deleted, just nulled out. The deletion is tied to the > erasure from the list (because the list holds the entry with a unique_ptr<>) so > in theory it should always be undeleted while its on the list and deleted after > it's off. Which means this is weird and "can't happen". (Though it might be a > wandering pointer bug, which would be all sorts of fun to track down.) > > A really large hammer: Create a data structure on HttpCache that is used to > store a large set of stack traces from deletion of an HttpCache::ActiveEntry. > Store the last 100 or something like that. Have that DS be looked up by > pointer. Erase elements if a new ActiveEntry is created with the same pointer. > Do an explicit crash if there's a second deletion of an active entry that's > already present in the set, after getting the stack trace from the first > deletion onto the stack. You'll then need to look at that stack in the minidump > with a debugger, but if the problem is indeed a free-after-free, that would get > you a long step towards figuring out what's going on. > > I'll keep musing on this a bit and see if I can come up with anything else to > suggest. > > > > FWIW, I don't like the pattern used in ~HttpCache around the active entries, or Please ignore this last line--I intended to remove it before sending. (I dislike the pattern "Delete, then remove from list", but this is the pattern "Null, then remove from list" which isn't my favorite, but isn't particularly bad.)
On 2017/05/11 at 18:38:02, rdsmith wrote: > On 2017/05/11 17:50:45, shivanisha wrote: > > For the canary (60.0.3082.0) crashes, all but one of them can be speculated to > > be fixed because of the fixes in patches 34 (Cached HEAD request handling fixed) > > and 35 (Fixed partial transactions handling in ActiveEntry). Most of them are > > related to the inconsistency where a transaction thinks it is associated with > > the entry while the entry does not have a reference to it (this happened for > > partial requests) OR transaction assuming that it is a reader in the entry while > > it was the headers_transaction (this happened for HEAD requests). > > > > For the one report, that I am not able to put a root cause to, is in the > > shutdown case where HttpCache is trying to access a destroyed entry? > > > > Josh/Randy, Does something come in your mind regarding how this CL could be > > leading to this crash: example report ids are 5c8dbcee80000000 or > > 34c6757b30000000 or 6f014fd0d0000000 ? Thanks. > > If not, may be I'll wait to see if it gets fixed with the fixes now done in the > > canary when this lands. > > > > All of these crashes are at statement number 353 in http_cache.cc (see patch 33) > > entry->readers.clear(); > > Just to make sure I understand the details: What makes you say that it's a destroyed entry? Just that there's a problem on clearing a std:: container of pointers? (FWIW, I can't think of any other reason why that would crash other than use-after-free, but if you have other information confirming that, it's probably good that I understand it.) Good question. I assumed it would be a destroyed entry because of lack of a better explanation. One of the reports has an ASAN trace(https://crash.corp.google.com/browse?q=&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=&stbtiq=product:Chrome%205c8dbcee80000000&reportid=5c8dbcee80000000&index=0). The ASAN trace might be helpful in confirming if its indeed a destroyed entry read case or something else. I am not completely sure if the ASAN trace is mapping the correct symbols but its alloc and free stacks point to content::RenderFrameHostImpl::SetUpMojoIfNeeded() and ~ServiceManagerConnectionImpl() respectively.
https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... File net/http/http_transaction.h (right): https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... net/http/http_transaction.h:56: // the consumer is alive; after that point, the HttpTransaction will not On 2017/05/10 at 21:34:26, Randy Smith (Not in Mondays) wrote: > On 2017/05/10 14:46:03, shivanisha wrote: > > On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > > > How does the HttpTransaction know when its consumer dies? > > > > > > (I had thought we kept the API contract to being "the HttpTransaction will be > > destroyed when its consumer is destroyed"; i.e. the HttpTransaction would be > > owned by its consumer. > > In that case, I think this requirement translates to "The consumer must ensure > > that |*request_info| remains alive for the lifetime of the HttpTransaction.", > > which is both a simple and more common interface contract.) > > > > That's true for HttpCache::Transaction but not for HttpNetworkTransaction since > > when we land parallel writing, it will be owning HttpNetworkTransaction even > > after consumer has died. > > > > Ok, I think we're using terminology differently: When I say "consumer" for a particular network transaction, I mean the immediate class that's driving it. So the consumer for the HttpNetworkTransaction would be the owning HttpCacheTransaction. > > If the lifetime for |*request_info| is the lifetime of the owning URLRequestHttpJob, then my concern is even stronger. How is the HttpNetworkTransaction going to know when that dies? If it can't, how can this interface contract be useful for the HttpNetworkTransaction if it can't know? > > I'm inclined to think that interface contracts with regard to time should be written s.t. both the (immediate) consumer and implementing class have access to the event occurrences that are mentioned. Otherwise it doesn't really make sense as an interface contract. But I suspect there's something I'm missing about event visibility here. Agree with consumer being the immediate consumer which means HttpCache::Transaction for HttpNetworkTransaction in a typical case. HttpNetworkTransaction will not know that its consumer that passed the request_info to it in the first place has died, but it makes an approximation by nulling the request info pointer when headers are received. That made the comment refer to headers in the first place. But then for another derived class of HttpTransaction, HttpCache::Transaction, headers may be received multiple time (for partial requests) and thus the headers boundary does not apply to it. So the newest comment in a way wants to tell the HttpTransaction in question to make an inference if it thinks its initial consumer might die before it and null the request info ptr accordingly at the right point. Since HC::T knows that it will not outlive its initial consumer , it doesn't really need to null the request info pointer. https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.c... net/http/http_cache.cc:847: CHECK(is_partial || (!entry->writer && entry->done_headers_queue.empty())); On 2017/05/10 at 21:34:26, Randy Smith (Not in Mondays) wrote: > On 2017/05/10 14:46:03, shivanisha wrote: > > On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > > > This is a (check-level) behavior change. Crashes used to happen if > > |is_partial && !entry->done_headers_queue.empty()| and no longer will. The > > description of this PS didn't mention a change in check behavior, just changing > > from DCHECK->CHECK. Why the change? > > > > Ah right. I forgot to mention that in the patch description. The patch that > > changed the behavior of the dcheck to include partial cases was originally patch > > 35, but it was incomplete because it made !entry->writer true only for > > non-partial requests but not entry->done_headers_queue.empty(). Basically, in > > case of partial , since any request can have write mode bit set irrespective of > > whether there is a writer or not or whether there is a done_headers_queue > > transaction or not. > > Gotcha; that makes sense. > > > Responding to the high level question here as well, my intention here was to > > make these assertions as CHECKs permanently since these are important invariants > > of the code and if they ever break in production , they should be easy to see > > from the backtrace. I did find a few more CHECK assertions in net/. Let me know > > if that's a concern. > > Hmmm. I don't think of that as the normal idiom. I think of CHECKs being in production code in two cases: a) We're doing a run against canary to figure out where a particular failure is really happening, and b) the consequences of the CHECK not being there and the code continuing to execute are clearly so bad that it's better for the end user that the browser crash than that we blow through the CHECK. I don't think either applies here. > > I don't claim to be a complete authority on this issue--if you wanted to run it by Matt, I'd happily accept his override. But my call would be that these CHECKs shouldn't be in on a long-term basis. a) is true because speculatively most of canary crashes were because of these assertions not being true and I would like to see if that's the case. In case the crashes are still there with these CHECKs, that would lead to a better investigation. b) seems true because these checks assert that both the HC::T and entry are on the same page. If not, that is bad for the code to proceed. But I am also alright with adding a TODO to remove them eventually.
On 2017/05/11 19:01:23, shivanisha wrote: > On 2017/05/11 at 18:38:02, rdsmith wrote: > > On 2017/05/11 17:50:45, shivanisha wrote: > > > For the canary (60.0.3082.0) crashes, all but one of them can be speculated > to > > > be fixed because of the fixes in patches 34 (Cached HEAD request handling > fixed) > > > and 35 (Fixed partial transactions handling in ActiveEntry). Most of them > are > > > related to the inconsistency where a transaction thinks it is associated > with > > > the entry while the entry does not have a reference to it (this happened for > > > partial requests) OR transaction assuming that it is a reader in the entry > while > > > it was the headers_transaction (this happened for HEAD requests). > > > > > > For the one report, that I am not able to put a root cause to, is in the > > > shutdown case where HttpCache is trying to access a destroyed entry? > > > > > > Josh/Randy, Does something come in your mind regarding how this CL could be > > > leading to this crash: example report ids are 5c8dbcee80000000 or > > > 34c6757b30000000 or 6f014fd0d0000000 ? Thanks. > > > If not, may be I'll wait to see if it gets fixed with the fixes now done in > the > > > canary when this lands. > > > > > > All of these crashes are at statement number 353 in http_cache.cc (see patch > 33) > > > entry->readers.clear(); > > > > Just to make sure I understand the details: What makes you say that it's a > destroyed entry? Just that there's a problem on clearing a std:: container of > pointers? (FWIW, I can't think of any other reason why that would crash other > than use-after-free, but if you have other information confirming that, it's > probably good that I understand it.) > > Good question. I assumed it would be a destroyed entry because of lack of a > better explanation. > > One of the reports has an ASAN > trace(https://crash.corp.google.com/browse?q=&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=&stbtiq=product:Chrome%205c8dbcee80000000&reportid=5c8dbcee80000000&index=0). > The ASAN trace might be helpful in confirming if its indeed a destroyed entry > read case or something else. I am not completely sure if the ASAN trace is > mapping the correct symbols but its alloc and free stacks point to > content::RenderFrameHostImpl::SetUpMojoIfNeeded() and > ~ServiceManagerConnectionImpl() respectively. I'm not familiar with Asan traces; are those traces intended to show the last alloc/free for the memory area where the crash happened? That would suggest that we're very much seeing a use-after-free.
On 2017/05/11 at 20:40:25, rdsmith wrote: > On 2017/05/11 19:01:23, shivanisha wrote: > > On 2017/05/11 at 18:38:02, rdsmith wrote: > > > On 2017/05/11 17:50:45, shivanisha wrote: > > > > For the canary (60.0.3082.0) crashes, all but one of them can be speculated > > to > > > > be fixed because of the fixes in patches 34 (Cached HEAD request handling > > fixed) > > > > and 35 (Fixed partial transactions handling in ActiveEntry). Most of them > > are > > > > related to the inconsistency where a transaction thinks it is associated > > with > > > > the entry while the entry does not have a reference to it (this happened for > > > > partial requests) OR transaction assuming that it is a reader in the entry > > while > > > > it was the headers_transaction (this happened for HEAD requests). > > > > > > > > For the one report, that I am not able to put a root cause to, is in the > > > > shutdown case where HttpCache is trying to access a destroyed entry? > > > > > > > > Josh/Randy, Does something come in your mind regarding how this CL could be > > > > leading to this crash: example report ids are 5c8dbcee80000000 or > > > > 34c6757b30000000 or 6f014fd0d0000000 ? Thanks. > > > > If not, may be I'll wait to see if it gets fixed with the fixes now done in > > the > > > > canary when this lands. > > > > > > > > All of these crashes are at statement number 353 in http_cache.cc (see patch > > 33) > > > > entry->readers.clear(); > > > > > > Just to make sure I understand the details: What makes you say that it's a > > destroyed entry? Just that there's a problem on clearing a std:: container of > > pointers? (FWIW, I can't think of any other reason why that would crash other > > than use-after-free, but if you have other information confirming that, it's > > probably good that I understand it.) > > > > Good question. I assumed it would be a destroyed entry because of lack of a > > better explanation. > > > > One of the reports has an ASAN > > trace(https://crash.corp.google.com/browse?q=&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=&stbtiq=product:Chrome%205c8dbcee80000000&reportid=5c8dbcee80000000&index=0). > > The ASAN trace might be helpful in confirming if its indeed a destroyed entry > > read case or something else. I am not completely sure if the ASAN trace is > > mapping the correct symbols but its alloc and free stacks point to > > content::RenderFrameHostImpl::SetUpMojoIfNeeded() and > > ~ServiceManagerConnectionImpl() respectively. > > I'm not familiar with Asan traces; are those traces intended to show the last alloc/free for the memory area where the crash happened? That would suggest that we're very much seeing a use-after-free. Yes, it does mention the error as heap-use-after-free but as per the traces it does not look like it is accessing freed entry, probably freed HttpCache? Not sure. Yes, the asan traces show the use stack , allocated stack and free stack for that pointer.
On 2017/05/11 20:45:43, shivanisha wrote: > On 2017/05/11 at 20:40:25, rdsmith wrote: > > On 2017/05/11 19:01:23, shivanisha wrote: > > > On 2017/05/11 at 18:38:02, rdsmith wrote: > > > > On 2017/05/11 17:50:45, shivanisha wrote: > > > > > For the canary (60.0.3082.0) crashes, all but one of them can be > speculated > > > to > > > > > be fixed because of the fixes in patches 34 (Cached HEAD request > handling > > > fixed) > > > > > and 35 (Fixed partial transactions handling in ActiveEntry). Most of > them > > > are > > > > > related to the inconsistency where a transaction thinks it is associated > > > with > > > > > the entry while the entry does not have a reference to it (this happened > for > > > > > partial requests) OR transaction assuming that it is a reader in the > entry > > > while > > > > > it was the headers_transaction (this happened for HEAD requests). > > > > > > > > > > For the one report, that I am not able to put a root cause to, is in the > > > > > shutdown case where HttpCache is trying to access a destroyed entry? > > > > > > > > > > Josh/Randy, Does something come in your mind regarding how this CL could > be > > > > > leading to this crash: example report ids are 5c8dbcee80000000 or > > > > > 34c6757b30000000 or 6f014fd0d0000000 ? Thanks. > > > > > If not, may be I'll wait to see if it gets fixed with the fixes now done > in > > > the > > > > > canary when this lands. > > > > > > > > > > All of these crashes are at statement number 353 in http_cache.cc (see > patch > > > 33) > > > > > entry->readers.clear(); > > > > > > > > Just to make sure I understand the details: What makes you say that it's a > > > destroyed entry? Just that there's a problem on clearing a std:: container > of > > > pointers? (FWIW, I can't think of any other reason why that would crash > other > > > than use-after-free, but if you have other information confirming that, it's > > > probably good that I understand it.) > > > > > > Good question. I assumed it would be a destroyed entry because of lack of a > > > better explanation. > > > > > > One of the reports has an ASAN > > > > trace(https://crash.corp.google.com/browse?q=&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=&stbtiq=product:Chrome%205c8dbcee80000000&reportid=5c8dbcee80000000&index=0). > > > The ASAN trace might be helpful in confirming if its indeed a destroyed > entry > > > read case or something else. I am not completely sure if the ASAN trace is > > > mapping the correct symbols but its alloc and free stacks point to > > > content::RenderFrameHostImpl::SetUpMojoIfNeeded() and > > > ~ServiceManagerConnectionImpl() respectively. > > > > I'm not familiar with Asan traces; are those traces intended to show the last > alloc/free for the memory area where the crash happened? That would suggest > that we're very much seeing a use-after-free. > > Yes, it does mention the error as heap-use-after-free but as per the traces it > does not look like it is accessing freed entry, probably freed HttpCache? Not > sure. > Yes, the asan traces show the use stack , allocated stack and free stack for > that pointer. My naive assumption is that the HttpCache freed the pointer, it was then allocated by something else (from the viewpoint of ActiveEntry, scrambling the memory) and then freed. However, that theory doesn't account for the alloc/free being shown not matching--that would seem to imply yet another alloc/free mismatch, which seems unlikely enough that I presume I'm not understanding what the trace is trying to tell me :-J.
https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... File net/http/http_transaction.h (right): https://codereview.chromium.org/2721933002/diff/960001/net/http/http_transact... net/http/http_transaction.h:56: // the consumer is alive; after that point, the HttpTransaction will not On 2017/05/11 19:58:24, shivanisha wrote: > On 2017/05/10 at 21:34:26, Randy Smith (Not in Mondays) wrote: > > On 2017/05/10 14:46:03, shivanisha wrote: > > > On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > > > > How does the HttpTransaction know when its consumer dies? > > > > > > > > (I had thought we kept the API contract to being "the HttpTransaction will > be > > > destroyed when its consumer is destroyed"; i.e. the HttpTransaction would be > > > owned by its consumer. > > > In that case, I think this requirement translates to "The consumer must > ensure > > > that |*request_info| remains alive for the lifetime of the > HttpTransaction.", > > > which is both a simple and more common interface contract.) > > > > > > That's true for HttpCache::Transaction but not for HttpNetworkTransaction > since > > > when we land parallel writing, it will be owning HttpNetworkTransaction even > > > after consumer has died. > > > > > > > Ok, I think we're using terminology differently: When I say "consumer" for a > particular network transaction, I mean the immediate class that's driving it. > So the consumer for the HttpNetworkTransaction would be the owning > HttpCacheTransaction. > > > > If the lifetime for |*request_info| is the lifetime of the owning > URLRequestHttpJob, then my concern is even stronger. How is the > HttpNetworkTransaction going to know when that dies? If it can't, how can this > interface contract be useful for the HttpNetworkTransaction if it can't know? > > > > I'm inclined to think that interface contracts with regard to time should be > written s.t. both the (immediate) consumer and implementing class have access to > the event occurrences that are mentioned. Otherwise it doesn't really make > sense as an interface contract. But I suspect there's something I'm missing > about event visibility here. > > Agree with consumer being the immediate consumer which means > HttpCache::Transaction for HttpNetworkTransaction in a typical case. > HttpNetworkTransaction will not know that its consumer that passed the > request_info to it in the first place has died, but it makes an approximation by > nulling the request info pointer when headers are received. That made the > comment refer to headers in the first place. But then for another derived class > of HttpTransaction, HttpCache::Transaction, headers may be received multiple > time (for partial requests) and thus the headers boundary does not apply to it. > So the newest comment in a way wants to tell the HttpTransaction in question to > make an inference if it thinks its initial consumer might die before it and null > the request info ptr accordingly at the right point. > Since HC::T knows that it will not outlive its initial consumer , it doesn't > really need to null the request info pointer. First of all, remind me what the intended use case is for this bit of interface contract? I'm remembering doing this when you were going to orphan HttpCache::Transactions to allow them to drive HttpNetworkTransaction after their owner (URLRequestHttpJob) had died, but I thought we've moved away from that design. Unless we have a specific current or soon expected use case in which an HttpTransaction will outlive its consumer, I really think this should just be "The consumer must ensure that |*request_info| outlives the HttpTransaction." If we do have such a use case, I think this needs to be phrased in terms of what the requirements on the consumer are s.t. the transaction will be able to access |*request_info| appropriately. This comment is tricky, because it's specifying requirements for *both* the consumer and for the actual subclass implementing HttpTransaction. But that means that what it specifies needs to be visible to both of those targets. Since the HttpTransaction doesn't have visibility into when it's consumer dies, I don't like the current spec. https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.c... net/http/http_cache.cc:847: CHECK(is_partial || (!entry->writer && entry->done_headers_queue.empty())); On 2017/05/11 19:58:24, shivanisha wrote: > On 2017/05/10 at 21:34:26, Randy Smith (Not in Mondays) wrote: > > On 2017/05/10 14:46:03, shivanisha wrote: > > > On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > > > > This is a (check-level) behavior change. Crashes used to happen if > > > |is_partial && !entry->done_headers_queue.empty()| and no longer will. The > > > description of this PS didn't mention a change in check behavior, just > changing > > > from DCHECK->CHECK. Why the change? > > > > > > Ah right. I forgot to mention that in the patch description. The patch that > > > changed the behavior of the dcheck to include partial cases was originally > patch > > > 35, but it was incomplete because it made !entry->writer true only for > > > non-partial requests but not entry->done_headers_queue.empty(). Basically, > in > > > case of partial , since any request can have write mode bit set irrespective > of > > > whether there is a writer or not or whether there is a done_headers_queue > > > transaction or not. > > > > Gotcha; that makes sense. > > > > > Responding to the high level question here as well, my intention here was to > > > make these assertions as CHECKs permanently since these are important > invariants > > > of the code and if they ever break in production , they should be easy to > see > > > from the backtrace. I did find a few more CHECK assertions in net/. Let me > know > > > if that's a concern. > > > > Hmmm. I don't think of that as the normal idiom. I think of CHECKs being in > production code in two cases: a) We're doing a run against canary to figure out > where a particular failure is really happening, and b) the consequences of the > CHECK not being there and the code continuing to execute are clearly so bad that > it's better for the end user that the browser crash than that we blow through > the CHECK. I don't think either applies here. > > > > I don't claim to be a complete authority on this issue--if you wanted to run > it by Matt, I'd happily accept his override. But my call would be that these > CHECKs shouldn't be in on a long-term basis. > > a) is true because speculatively most of canary crashes were because of these > assertions not being true and I would like to see if that's the case. In case > the crashes are still there with these CHECKs, that would lead to a better > investigation. I'm perfectly happy with you having CHECKs in to help diagnose canary crashes as long as there's a bug-linked TODO to remove them when the bug is fixed. > b) seems true because these checks assert that both the HC::T and entry are on > the same page. If not, that is bad for the code to proceed. We differ here. It's a matter of relative badness--yes, the code proceeding could result in ugly stuff happening. But crashing will *definitely* result in ugly stuff happening. And once things are in the field, the user experience is (IMO) the right metric. So you have to believe that proceeding makes it likely that, from the user's perspective, something worse than a browser crash will occur before a CHECK() is appropriate. There isn't a lot worse from the user's perspective than a browser crash. Another way to think of it is that, if you have a CHECK in the code and you push it to stable, then you're using that CHECK() as an error handling mechanism, and it's actually a pretty user-unfriendly error handling mechanism. That doesn't mean it's always wrong--if something is really unlikely it may not be worth the effort to put in more user friendly error handling, and if something is really bad it may be worth crashing the browser rather than letting it happen. But for me "really bad" means "Will expose cookies to untrusted third parties" or "may create a security vulnerability allowing system perversion", not "Might crash later because data structures get inconsistent". But as I say, I don't consider myself perfectly grounded in what our guidelines are in this space. But if you want to go with consulting with just me, put in a bug-linked TODO and put them back to DCHECKs after canary is clean. > > But I am also alright with adding a TODO to remove them eventually.
https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1000001/net/http/http_cache.c... net/http/http_cache.cc:847: CHECK(is_partial || (!entry->writer && entry->done_headers_queue.empty())); On 2017/05/11 21:47:09, Randy Smith (Not in Mondays) wrote: > On 2017/05/11 19:58:24, shivanisha wrote: > > On 2017/05/10 at 21:34:26, Randy Smith (Not in Mondays) wrote: > > > On 2017/05/10 14:46:03, shivanisha wrote: > > > > On 2017/05/09 at 23:20:32, Randy Smith (Not in Mondays) wrote: > > > > > This is a (check-level) behavior change. Crashes used to happen if > > > > |is_partial && !entry->done_headers_queue.empty()| and no longer will. > The > > > > description of this PS didn't mention a change in check behavior, just > > changing > > > > from DCHECK->CHECK. Why the change? > > > > > > > > Ah right. I forgot to mention that in the patch description. The patch > that > > > > changed the behavior of the dcheck to include partial cases was originally > > patch > > > > 35, but it was incomplete because it made !entry->writer true only for > > > > non-partial requests but not entry->done_headers_queue.empty(). Basically, > > in > > > > case of partial , since any request can have write mode bit set > irrespective > > of > > > > whether there is a writer or not or whether there is a done_headers_queue > > > > transaction or not. > > > > > > Gotcha; that makes sense. > > > > > > > Responding to the high level question here as well, my intention here was > to > > > > make these assertions as CHECKs permanently since these are important > > invariants > > > > of the code and if they ever break in production , they should be easy to > > see > > > > from the backtrace. I did find a few more CHECK assertions in net/. Let me > > know > > > > if that's a concern. > > > > > > Hmmm. I don't think of that as the normal idiom. I think of CHECKs being > in > > production code in two cases: a) We're doing a run against canary to figure > out > > where a particular failure is really happening, and b) the consequences of the > > CHECK not being there and the code continuing to execute are clearly so bad > that > > it's better for the end user that the browser crash than that we blow through > > the CHECK. I don't think either applies here. > > > > > > I don't claim to be a complete authority on this issue--if you wanted to run > > it by Matt, I'd happily accept his override. But my call would be that these > > CHECKs shouldn't be in on a long-term basis. > > > > a) is true because speculatively most of canary crashes were because of these > > assertions not being true and I would like to see if that's the case. In case > > the crashes are still there with these CHECKs, that would lead to a better > > investigation. > > I'm perfectly happy with you having CHECKs in to help diagnose canary crashes as > long as there's a bug-linked TODO to remove them when the bug is fixed. > > > b) seems true because these checks assert that both the HC::T and entry are on > > the same page. If not, that is bad for the code to proceed. > > We differ here. It's a matter of relative badness--yes, the code proceeding > could result in ugly stuff happening. But crashing will *definitely* result in > ugly stuff happening. And once things are in the field, the user experience is > (IMO) the right metric. So you have to believe that proceeding makes it likely > that, from the user's perspective, something worse than a browser crash will > occur before a CHECK() is appropriate. There isn't a lot worse from the user's > perspective than a browser crash. > > Another way to think of it is that, if you have a CHECK in the code and you push > it to stable, then you're using that CHECK() as an error handling mechanism, and > it's actually a pretty user-unfriendly error handling mechanism. That doesn't > mean it's always wrong--if something is really unlikely it may not be worth the > effort to put in more user friendly error handling, and if something is really > bad it may be worth crashing the browser rather than letting it happen. But for > me "really bad" means "Will expose cookies to untrusted third parties" or "may > create a security vulnerability allowing system perversion", not "Might crash > later because data structures get inconsistent". > > But as I say, I don't consider myself perfectly grounded in what our guidelines > are in this space. But if you want to go with consulting with just me, put in a > bug-linked TODO and put them back to DCHECKs after canary is clean. > > > > > But I am also alright with adding a TODO to remove them eventually. > The style guide makes the usage of CHECKS clear: Use CHECK() if the consequence of a failed assertion would be a security vulnerability, where crashing the browser is preferable. Because this takes down the whole browser, sometimes there are better options than CHECK(). For example, if a renderer sends the browser process a malformed IPC, an attacker may control the renderer, but we can simply kill the offending renderer instead of crashing the whole browser. You can temporarily use CHECK() instead of DCHECK() when trying to force crashes in release builds to sniff out which of your assertions is failing. Donāt leave these in the codebase forever; remove them or change them back once youāve solved the problem.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
PTAL, Thanks. The latest patch adds the TODOs for converting CHECKs to DCHECKs and for changing the interface of HttpTransaction::Start w.r.t Request Info.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: win_chromium_x64_rel_ng on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
PTAL, Thanks! The latest patch converts the calls to DoneReadingFromEntry to DoneWithEntry on sites where it cannot be ensured that the transaction is actually a reader. Earlier READ mode implied reader in these sites but not anymore since a transaction's READ mode is set first along with adding it to done_headers_queue and a task is posted. On processing that task it will be added to readers. I saw this issue very rarely in a DCHECK getting hit in DoneReading->DoneReadingFromEntry. It was a race and wasn't reproducible, and seems to have gotten fixed with this change.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: win_chromium_x64_rel_ng on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)
LGTM modulo linking TODOs to an actual bug for hopefully not happening future archaeology. https://codereview.chromium.org/2721933002/diff/1060001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1060001/net/http/http_cache.c... net/http/http_cache.cc:835: // canary is clear of any crashes. Could you make these TODOs linked to a particular bug you're trying to fix? What I don't want is for these TODOs to accidentally get left in the code and someone to come on them later and have no way to figure out if it's possible to shift them back. https://codereview.chromium.org/2721933002/diff/1060001/net/url_request/url_r... File net/url_request/url_request_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/1060001/net/url_request/url_r... net/url_request/url_request_unittest.cc:1200: &d, TRAFFIC_ANNOTATION_FOR_TESTS)); Note for future CLs/PSs: I presume this means that this PS contains both your changes and a resync to a different backing commit on the main tree. It's much easier for reviewers if those two things happen in two separate CLs, that way if they're reviewing by patchset, they can just focus on your changes and ignore what's happened on the main trunk. https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache.c... net/http/http_cache.cc:867: void HttpCache::DoneWithEntry(ActiveEntry* entry, Suggestion: Given that this routine appears to do the right thing with all transactions and that there's only one call to DoneReadingFromEntry left in http_cache_transaction.cc, would it make sense to switch that last call and make DoneReadingFromEntry private on HttpCache? Or even just pull the code into DoneWithEntry()? https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache.c... net/http/http_cache.cc:913: DoneReadingFromEntry(entry, transaction); Suggestion, low priority: Is it possible to put a DCHECK here to confirm the transaction's mode is READ?
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Rebased. https://codereview.chromium.org/2721933002/diff/1100001/net/http/http_cache_u... File net/http/http_cache_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/1100001/net/http/http_cache_u... net/http/http_cache_unittest.cc:1193: Starting from here till my next comment, these tests weren't really added by my CL and looks to be the same as in master. Not sure why are they being shown as new additions in the diff. https://codereview.chromium.org/2721933002/diff/1100001/net/http/http_cache_u... net/http/http_cache_unittest.cc:1355: Starting from my last comment till here, these tests weren't really added by my CL and looks to be the same as in master. Not sure why are they being shown as new additions in the diff. Pointing this out so you know what to review in this file.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
PTAL, Thanks! https://codereview.chromium.org/2721933002/diff/1060001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1060001/net/http/http_cache.c... net/http/http_cache.cc:835: // canary is clear of any crashes. On 2017/05/21 at 21:32:02, Randy Smith (Not in Mondays) wrote: > Could you make these TODOs linked to a particular bug you're trying to fix? What I don't want is for these TODOs to accidentally get left in the code and someone to come on them later and have no way to figure out if it's possible to shift them back. Done https://codereview.chromium.org/2721933002/diff/1060001/net/url_request/url_r... File net/url_request/url_request_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/1060001/net/url_request/url_r... net/url_request/url_request_unittest.cc:1200: &d, TRAFFIC_ANNOTATION_FOR_TESTS)); On 2017/05/21 at 21:32:02, Randy Smith (Not in Mondays) wrote: > Note for future CLs/PSs: I presume this means that this PS contains both your changes and a resync to a different backing commit on the main tree. It's much easier for reviewers if those two things happen in two separate CLs, that way if they're reviewing by patchset, they can just focus on your changes and ignore what's happened on the main trunk. Acknowledged. Sorry for any confusions. https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache.c... net/http/http_cache.cc:867: void HttpCache::DoneWithEntry(ActiveEntry* entry, On 2017/05/21 at 21:32:02, Randy Smith (Not in Mondays) wrote: > Suggestion: Given that this routine appears to do the right thing with all transactions and that there's only one call to DoneReadingFromEntry left in http_cache_transaction.cc, would it make sense to switch that last call and make DoneReadingFromEntry private on HttpCache? Or even just pull the code into DoneWithEntry()? I am inclined to leave the last call in HttpCache::Transaction::DoCacheReadDataComplete to be as is, for now since that will assert that only readers can be in that state. Note that , it will need to be changed once parallel writing is implemented. https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache.c... net/http/http_cache.cc:913: DoneReadingFromEntry(entry, transaction); On 2017/05/21 at 21:32:02, Randy Smith (Not in Mondays) wrote: > Suggestion, low priority: Is it possible to put a DCHECK here to confirm the transaction's mode is READ? I am slightly nervous about putting that DCHECK here because of partial transactions. I am worried there will be some partial transaction's flow that will not comply with that assumption.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...) ios-simulator on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/bui...)
Still LGTM. Thanks!
Note that the comments are spread over two patchsets. https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache_t... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache_t... net/http/http_cache_transaction.cc:199: cache_->DoneWithEntry(entry_, this, true, partial_ != nullptr); Yay for this cleanup! https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache_t... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache_t... net/http/http_cache_transaction.h:187: // Used when restarting the transaction. The comment could be more clear about where this data is coming from, for example: "A snapshot of pieces of the transaction before entering the state machine so that the state can be restored when restarting the state machine." https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache_t... net/http/http_cache_transaction.h:542: RestartInfo restart_info_; Please add a comment to this. https://codereview.chromium.org/2721933002/diff/1120001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1120001/net/http/http_cache.c... net/http/http_cache.cc:876: entry->disk_entry->CancelSparseIO(); What if we didn't call CancelSparseIO? Would anything bad happen? From the description it doesn't seem like it. Then we could remove the logically complex IsTransactionExclusivelyWriting function. https://codereview.chromium.org/2721933002/diff/1120001/net/http/http_cache_t... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/1120001/net/http/http_cache_t... net/http/http_cache_transaction.cc:1385: cache_->DoneWithEntry(entry_, this, false, partial_ != nullptr); Can you add a /* */ comment describing what argument false is?
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Patchset #49 (id:1140001) has been deleted
jkarlin, PTAL, Thanks! The latest patch has feedback addressed + a dcheck removed. The dcheck assumed that if we are in DoPartialHeadersReceived() with partial_ not set, then we can assume it is a full request and thus can assert that it is in headers phase with DCHECK(!reading_).Turns out, this state is also reached in truncate flow which can happen any time, even after reading has started and when the transaction is destroyed. https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache_t... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache_t... net/http/http_cache_transaction.h:187: // Used when restarting the transaction. On 2017/05/30 at 14:42:38, jkarlin wrote: > The comment could be more clear about where this data is coming from, for example: > > "A snapshot of pieces of the transaction before entering the state machine so that the state can be restored when restarting the state machine." Done https://codereview.chromium.org/2721933002/diff/1080001/net/http/http_cache_t... net/http/http_cache_transaction.h:542: RestartInfo restart_info_; On 2017/05/30 at 14:42:38, jkarlin wrote: > Please add a comment to this. done https://codereview.chromium.org/2721933002/diff/1120001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1120001/net/http/http_cache.c... net/http/http_cache.cc:876: entry->disk_entry->CancelSparseIO(); On 2017/05/30 at 14:42:38, jkarlin wrote: > What if we didn't call CancelSparseIO? Would anything bad happen? From the description it doesn't seem like it. Then we could remove the logically complex IsTransactionExclusivelyWriting function. IsTransactionExclusivelyWriting's return value is not just used for deciding to invoke CancelSparseIO but also for other cases further down. https://codereview.chromium.org/2721933002/diff/1120001/net/http/http_cache_t... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/1120001/net/http/http_cache_t... net/http/http_cache_transaction.cc:1385: cache_->DoneWithEntry(entry_, this, false, partial_ != nullptr); On 2017/05/30 at 14:42:38, jkarlin wrote: > Can you add a /* */ comment describing what argument false is? done , here and in all invocation sites.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
https://codereview.chromium.org/2721933002/diff/1160001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1160001/net/http/http_cache.c... net/http/http_cache.cc:872: bool cancel = false; Suggest: s/cancel/should_restart/ bool should_restart = process_cancel && HasDependentTransactions(). https://codereview.chromium.org/2721933002/diff/1160001/net/http/http_cache.c... net/http/http_cache.cc:1059: bool HttpCache::IsTransactionExclusivelyWriting(ActiveEntry* entry, After F2F it looks like this function is trying to determine if any transactions in the ActiveEntry depend on this cancelled transaction completing successfully. If so, then at least those dependent transactions need to be restarted. So how about this for a simplification: 1. Rename to HasDependentTransactions() 2. The body becomes: return transaction->mode() & Transaction::WRITE && !TransactionInAddToEntryQueue(); We know that no transactions depend on something in add_to_entry. We also know that the transaction must be a writer to have dependencies.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Latest patch has josh's feedback addressed. https://codereview.chromium.org/2721933002/diff/1160001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1160001/net/http/http_cache.c... net/http/http_cache.cc:872: bool cancel = false; On 2017/06/01 at 17:22:38, jkarlin wrote: > Suggest: s/cancel/should_restart/ > > bool should_restart = process_cancel && HasDependentTransactions(). done
https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.c... net/http/http_cache.cc:891: // If the response is not written (cancel is true), consider it a failure. s/cancel/should_restart/ https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.c... net/http/http_cache.cc:909: DoneWritingToEntry(entry, success, transaction); Why do we pass success here? Doesn't this mean that so long as truncation worked then we're not going to restart the dependent requests? https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.c... net/http/http_cache.cc:1068: // that should still be considered a writer failure. Huh. What happens if the writer has StopCaching() called on it, but successfully completes (without finishing writing to the cache)? Are the read-only entries in done_headers_queue now in trouble? https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.c... net/http/http_cache.cc:1074: // If transaction is not in add_to_entry_queue and has a WRITE bit set, then WRITE bit set or is NONE, then there may be other...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Description was changed from ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. BUG=472740, 715913, 715974, 715920, 715911, Review-Url: https://codereview.chromium.org/2721933002 Cr-Commit-Position: refs/heads/master@{#467426} Committed: https://chromium.googlesource.com/chromium/src/+/1e2e347f957ef889aaee527bb757... ========== to ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. BUG=472740, 715913, 715974, 715920, 715911, 713348 Review-Url: https://codereview.chromium.org/2721933002 Cr-Commit-Position: refs/heads/master@{#467426} Committed: https://chromium.googlesource.com/chromium/src/+/1e2e347f957ef889aaee527bb757... ==========
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Patchset #51 (id:1200001) has been deleted
PTAL, Thanks! Latest patch fixes truncation on destruction and StopCaching flows. https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.c... net/http/http_cache.cc:891: // If the response is not written (cancel is true), consider it a failure. On 2017/06/01 at 18:49:21, jkarlin wrote: > s/cancel/should_restart/ done https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.c... net/http/http_cache.cc:909: DoneWritingToEntry(entry, success, transaction); On 2017/06/01 at 18:49:21, jkarlin wrote: > Why do we pass success here? Doesn't this mean that so long as truncation worked then we're not going to restart the dependent requests? You are right, done_headers_queue transactions did not see the response as truncated when they read the headers so they should be restarted. Added the functionality. Since these restarted transactions and any add_to_entry_queue transactions should be able to read the truncated entry, this entry is not doomed/destroyed. https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.c... net/http/http_cache.cc:1068: // that should still be considered a writer failure. On 2017/06/01 at 18:49:21, jkarlin wrote: > Huh. What happens if the writer has StopCaching() called on it, but successfully completes (without finishing writing to the cache)? > > Are the read-only entries in done_headers_queue now in trouble? Current behavior: When the transaction successfully completes reading the response, in the destructor it is marked as truncated and thus all following transactions know what to expect. Now that there are done_headers_queue, if StopCaching() is invoked on the writer , it should inform the entry so it can restart the dependent transactions and doom this entry. Added a call to cache_->DoneWritingToEntry from StopCaching(). Though this will lead to the original entry not being used but currently this does not impact any scenario since StopCaching() is immediately invoked after receiving headers. Also, this allows no future or simultaneous transactions to be blocked on the first transaction if its not caching. (This also resolves crbug.com/713348). https://codereview.chromium.org/2721933002/diff/1180001/net/http/http_cache.c... net/http/http_cache.cc:1074: // If transaction is not in add_to_entry_queue and has a WRITE bit set, then On 2017/06/01 at 18:49:21, jkarlin wrote: > WRITE bit set or is NONE, then there may be other... done
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
Looking good! Just a few minor comments. https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:815: // Always add a new transaction to the queue to maintain FIFO order. I don't understand this comment. https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:827: if (entry->writer == transaction) Can we DCHECK that this is a range request? https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:993: it = entry->done_headers_queue.erase(it); DCHECK_NE(transaction, done_headers_transaction); https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:999: Transaction* transaction) { Transaction argument is unused, but: DCHECK_NE(entry->headers_transaction, transaction); would be good https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:1089: // that should still be considered a writer failure. This below conditional is hard for me to parse. Could you add a bool to represent the positive? // Transaction's mode may have been set to NONE if StopCaching was invoked but that should still be considered a writing failure. bool writing_transaction = transaction->mode() & Transaction::WRITE || transaction->mode() == Transaction::None; if (!writing_transaction) return false; https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.h... net/http/http_cache.h:437: // headers_transactions is awaiting an asynchronous operation completion, s/headers_transactions/headers_transaction/ https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.h... net/http/http_cache.h:464: // on |transaction| completely writing the response. "about to start writing response body" isn't well defined. How about, Returns true if any transactions in the ActiveEntry depend on this transaction to complete writing to the cache. https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.cc:411: // this entry be doomed. transactions are restarted and this entry is doomed https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.cc:412: cache_->DoneWritingToEntry(entry_, false, this); DoneWritingToEntry(entry_, false /* success */, this) https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.cc:412: cache_->DoneWritingToEntry(entry_, false, this); It's seems possible that DoneWritingToEntry will be called later down the line for this same transaction. Can DoneWritingToEntry handle being called twice for the same transaction? https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.cc:565: DCHECK(!reading_); Hmm, perhaps also DCHECK_NE(STATE_UNSET, next_state_) just to help ensure that this transaction is waiting for a callback when this is called. https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.h:110: // completely written, |*is_truncated| will be set if it was actually It'd be cleaner if is_truncated were assigned a value regardless. https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.h:112: bool AddTruncatedFlag(bool* is_truncated); optional nit: rename is_truncated to did_truncate https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_transac... File net/http/http_transaction_test_util.h (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_transac... net/http/http_transaction_test_util.h:283: CompletionCallback callback_; // used for pause and restart. s/callback_/resume_start_callback_/ https://codereview.chromium.org/2721933002/diff/1220001/net/http/mock_http_ca... File net/http/mock_http_cache.h (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/mock_http_ca... net/http/mock_http_cache.h:240: Remove newline https://codereview.chromium.org/2721933002/diff/1220001/net/url_request/url_r... File net/url_request/url_request_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/1220001/net/url_request/url_r... net/url_request/url_request_unittest.cc:4194: base::RunLoop().Run(); Why the change to this file?
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:815: // Always add a new transaction to the queue to maintain FIFO order. On 2017/06/08 at 16:14:09, jkarlin wrote: > I don't understand this comment. No transaction should bypass the queue because this queue helps in processing the transactions in FIFO order. https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:827: if (entry->writer == transaction) On 2017/06/08 at 16:14:09, jkarlin wrote: > Can we DCHECK that this is a range request? done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:993: it = entry->done_headers_queue.erase(it); On 2017/06/08 at 16:14:09, jkarlin wrote: > DCHECK_NE(transaction, done_headers_transaction); done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:999: Transaction* transaction) { On 2017/06/08 at 16:14:09, jkarlin wrote: > Transaction argument is unused, but: > > DCHECK_NE(entry->headers_transaction, transaction); > > would be good done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:1089: // that should still be considered a writer failure. On 2017/06/08 at 16:14:09, jkarlin wrote: > This below conditional is hard for me to parse. Could you add a bool to represent the positive? > > > // Transaction's mode may have been set to NONE if StopCaching was invoked but that should still be considered a writing failure. > bool writing_transaction = transaction->mode() & Transaction::WRITE || transaction->mode() == Transaction::None; > > if (!writing_transaction) > return false; done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.h File net/http/http_cache.h (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.h... net/http/http_cache.h:437: // headers_transactions is awaiting an asynchronous operation completion, On 2017/06/08 at 16:14:09, jkarlin wrote: > s/headers_transactions/headers_transaction/ done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.h... net/http/http_cache.h:464: // on |transaction| completely writing the response. On 2017/06/08 at 16:14:09, jkarlin wrote: > "about to start writing response body" isn't well defined. > > How about, Returns true if any transactions in the ActiveEntry depend on this transaction to complete writing to the cache. done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.cc:411: // this entry be doomed. On 2017/06/08 at 16:14:09, jkarlin wrote: > transactions are restarted and this entry is doomed done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.cc:412: cache_->DoneWritingToEntry(entry_, false, this); On 2017/06/08 at 16:14:09, jkarlin wrote: > It's seems possible that DoneWritingToEntry will be called later down the line for this same transaction. Can DoneWritingToEntry handle being called twice for the same transaction? Since entry_ is marked null here , DoneWriting* should not be called again. https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.cc:565: DCHECK(!reading_); On 2017/06/08 at 16:14:09, jkarlin wrote: > Hmm, perhaps also DCHECK_NE(STATE_UNSET, next_state_) just to help ensure that this transaction is waiting for a callback when this is called. done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... File net/http/http_cache_transaction.h (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.h:110: // completely written, |*is_truncated| will be set if it was actually On 2017/06/08 at 16:14:10, jkarlin wrote: > It'd be cleaner if is_truncated were assigned a value regardless. Done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.h:112: bool AddTruncatedFlag(bool* is_truncated); On 2017/06/08 at 16:14:09, jkarlin wrote: > optional nit: rename is_truncated to did_truncate done https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_transac... File net/http/http_transaction_test_util.h (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_transac... net/http/http_transaction_test_util.h:283: CompletionCallback callback_; // used for pause and restart. On 2017/06/08 at 16:14:10, jkarlin wrote: > s/callback_/resume_start_callback_/ done https://codereview.chromium.org/2721933002/diff/1220001/net/http/mock_http_ca... File net/http/mock_http_cache.h (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/mock_http_ca... net/http/mock_http_cache.h:240: On 2017/06/08 at 16:14:10, jkarlin wrote: > Remove newline newline because it applies to multiple functions below. https://codereview.chromium.org/2721933002/diff/1220001/net/url_request/url_r... File net/url_request/url_request_unittest.cc (right): https://codereview.chromium.org/2721933002/diff/1220001/net/url_request/url_r... net/url_request/url_request_unittest.cc:4194: base::RunLoop().Run(); On 2017/06/08 at 16:14:10, jkarlin wrote: > Why the change to this file? Because of addition of the transaction to add_to_entry_queue in all cases.
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
lgtm with nit https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:999: Transaction* transaction) { On 2017/06/08 18:26:01, shivanisha wrote: > On 2017/06/08 at 16:14:09, jkarlin wrote: > > Transaction argument is unused, but: > > > > DCHECK_NE(entry->headers_transaction, transaction); > > > > would be good > > done Hmm, actually, it doesn't make sense to keep an unused parameter around. We should just remove it, even though the dcheck is nice to have. https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... File net/http/http_cache_transaction.cc (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache_t... net/http/http_cache_transaction.cc:412: cache_->DoneWritingToEntry(entry_, false, this); On 2017/06/08 18:26:02, shivanisha wrote: > On 2017/06/08 at 16:14:09, jkarlin wrote: > > It's seems possible that DoneWritingToEntry will be called later down the line > for this same transaction. Can DoneWritingToEntry handle being called twice for > the same transaction? > > Since entry_ is marked null here , DoneWriting* should not be called again. Acknowledged.
The CQ bit was checked by shivanisha@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Thanks Josh and Randy! https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.cc File net/http/http_cache.cc (right): https://codereview.chromium.org/2721933002/diff/1220001/net/http/http_cache.c... net/http/http_cache.cc:999: Transaction* transaction) { On 2017/06/13 at 13:03:31, jkarlin wrote: > On 2017/06/08 18:26:01, shivanisha wrote: > > On 2017/06/08 at 16:14:09, jkarlin wrote: > > > Transaction argument is unused, but: > > > > > > DCHECK_NE(entry->headers_transaction, transaction); > > > > > > would be good > > > > done > > Hmm, actually, it doesn't make sense to keep an unused parameter around. We should just remove it, even though the dcheck is nice to have. removed.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
The CQ bit was checked by shivanisha@chromium.org
The patchset sent to the CQ was uploaded after l-g-t-m from rdsmith@chromium.org, jkarlin@chromium.org Link to the patchset: https://codereview.chromium.org/2721933002/#ps1280001 (title: "nit addressed")
CQ is trying da patch. Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
CQ is committing da patch. Bot data: {"patchset_id": 1280001, "attempt_start_ts": 1497396394819540, "parent_rev": "b1917aabe088ba6e547b77feba01ec587d111e43", "commit_rev": "8061c420676998bda77caa74581ea8061860f438"}
Message was sent while issue was closed.
Description was changed from ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. BUG=472740, 715913, 715974, 715920, 715911, 713348 Review-Url: https://codereview.chromium.org/2721933002 Cr-Commit-Position: refs/heads/master@{#467426} Committed: https://chromium.googlesource.com/chromium/src/+/1e2e347f957ef889aaee527bb757... ========== to ========== This CL is a precursor to allowing shared writing to fix cache lock. This CL allows transactions to continue to their validation phase even when another transaction is the active reader/writer. After the validation phase, if its a match the transaction might wait till the response is written to the cache by the active writer. If its not a match the transaction will doom the entry and go to the network. In a subsequent CL, the not matching case will create a new entry as well. BUG=472740, 715913, 715974, 715920, 715911, 713348 Review-Url: https://codereview.chromium.org/2721933002 Cr-Original-Commit-Position: refs/heads/master@{#467426} Committed: https://chromium.googlesource.com/chromium/src/+/1e2e347f957ef889aaee527bb757... Review-Url: https://codereview.chromium.org/2721933002 Cr-Commit-Position: refs/heads/master@{#479204} Committed: https://chromium.googlesource.com/chromium/src/+/8061c420676998bda77caa74581e... ==========
Message was sent while issue was closed.
Committed patchset #54 (id:1280001) as https://chromium.googlesource.com/chromium/src/+/8061c420676998bda77caa74581e... |