does openocd flush Icache when we set a breakpoint with 2020.03

[ElliotCheng]

it's found that when we use openocd to set breakpoint, info b shows the breakpoint is set, but actual memory is not overwritten with brk_s instructions.
suppose openocd doesn't flush icache; if that's the case, can we add autoflush like metaware debugger?

[abrodkin]

@EvgeniiDidin could you please take a look at this one?

[anthony-kolesov]

GDB writes breakpoints to the target only before program is resumed, therefore BRK_S is never visible when inspecting memory from the GDB itself - it shows user the actual program being debugged, not a temporarily state where breakpoints appear and disappear. What is more in BRK_S has to be removed anyway to let program step through it. I believe there is a GDB setting value that allows to change this behavior to force it to write BPs immediately, though I'm not sure it will change the memory view - the sensible thing for the debugger would still be to replace the BRK_S with the memory content that BRK_S replaced.

This doesn't relate to caches, because debugger writes and reads memory bypassing the caches, and BRK_S is also written the same way. So it is not really an OpenOCD issue but GDBs.

[olerem]

OpenOCD should take care to flash ICache to let the software BP be actually executed by the CPU. Otherwise the CPU is executing old state of ICache and in many cases the cache reality is different to the RAM state

[ElliotCheng]

with mdb we could see memory contents replaced with actual brk instructions; so gdb doesn't do this?
tricky thing reported is that: if we disable dcache from the very beginning, and then setup breakpoints, these breakpoints would be functional; but memory contents are still the same as original code, this matches @anthony-kolesov 's description about memory view.

[anthony-kolesov]

Yes, this is an intentional behavior by GDB. You can use set breakpoint always-inserted on to change this:

GDB> help set breakpoint always-inserted
Set mode for inserting breakpoints.
When this mode is on, breakpoints are inserted immediately as soon as
they're created, kept inserted even when execution stops, and removed
only when the user deletes them.  When this mode is off (the default),
breakpoints are inserted only when execution continues, and removed
when execution stops.

Because this behavior is in common GDB code, not ARC-specific, it is just a difference between debuggers, that users have to get accustomed for. As far as I see LLDB also follows the GDB approach (at least by default), BRK_S is not visible to the user:

(lldb) br set -n main 
Breakpoint 1: where = dumpbcr.hs44_base.elf`main + 18 at dumpbcr.c:84, address = 0x00000142
(lldb) disas -n main 
dumpbcr.hs44_base.elf`main:
    0x130 <+0>:  st.aw  %r13, [%sp, -12]
    0x134 <+4>:  st_s   %r14, [%sp, 4]
    0x136 <+6>:  st     %blink, [%sp, 8]
    0x13a <+10>: mov_s  %r14, 0
    0x13c <+12>: mov_s  %r13, 2147486640
    0x142 <+18>: ld.ab  %r0, [%r13, 8]            ; <========= Should be a BRK_S here.
    0x146 <+22>: lr     %r1, %r0
    0x14a <+26>: ld     %r2, [%r13, -4]
    0x14e <+30>: bl_s   0x100                     ; print_bcr at dumpbcr.c:69
    0x150 <+32>: add_s  %r14, %r14, 1
    0x152 <+34>: brlo   %r14, 50, 0x142           ; <+18> at dumpbcr.c:84
    0x156 <+38>: mov_s  %r0, 0
    0x158 <+40>: ld     %blink, [%sp, 8]
    0x15c <+44>: ld_s   %r14, [%sp, 4]
    0x15e <+46>: ld.ab  %r13, [%sp, 12]
    0x162 <+50>: j_s    [%blink]

[ElliotCheng]

this is helpful, so with that always inserted option there should be consistant behavior, no need to flush or disable cache. thx!
ticket could be closed.

[ElliotCheng]

set breakpoint always-inserted makes it work; doesn't need to flush or disable cache to make breakpoints visible. FYI.

[abrodkin]

@anthony-kolesov looks like there are too many levels of abstractions :)
As I understood @ElliotCheng, for the customer breakpoints really work only if:

• Instruction cache is disabled OR
• set breakpoint always-inserted is used in GDB

That's a bit weird as I know for sure on our boards we see breakpoints work normally w/o doing an either thing listed above.

@ElliotCheng MDB by default flushes & invalidates caches (both instruction & data caches, well instruction cache could not be flushed - it only gets invalidated) every now and then: when you set/remove BP, single-step etc. Thus it hides quite a few funny problems related to memory and cache coherency. So I would strongly suggest you try to disable that way too conservative MDB's behavior with -flush_dcache=off and see what kind of behavior you get.

Also, when you say "disable cache to make breakpoints visible" which cache you mean is being disabled: instruction, data or both?

[olerem]

See https://github.com/olerem/openocd/blob/master/src/target/armv7a_cache.c
something like this should be implemented for ARC

[anthony-kolesov]

@abrodkin, I don't see mention that BPs don't work - just that user expects that when they examine memory contents, through disassembler or otherwise, they would start seeing inserted BRK_S/BRK instructions, which is not the case, because GDB doesn't work this way by default. It would be bizarre if BPs would work if BP is set immediately, but doesn't work if it was written with a delay. The problem is that MDB works differently, hence user expectations are different.

On every memory access operation OpenOCD does validate whether cache should be flushed and invalidated for memory writes, and flushed for memory reads. If cache has been flushed/invalidated once after executed has stopped, then the flush/invalidation is not repeated, e.g.:
memory write: https://github.com/foss-for-synopsys-dwc-arc-processors/openocd/blob/master/src/target/arc_mem.c#L79
cache invalidation functions start here: https://github.com/foss-for-synopsys-dwc-arc-processors/openocd/blob/master/src/target/arc32.c#L425
Relevant commit ae855b0 has been implemented in 2014.

[abrodkin]

@ElliotCheng could you please elaborate a bit on what and how works/doesn't work for the customer? If indeed execution doesn't stop on a location where BP was set or it's just missing brk instruction in the memory which is observed by the user, while on execution BP gets hit and CPU gets halted?

[ElliotCheng]

@abrodkin sorry missed your replies. Here are the steps and results:

1. boot up with I$ enabled/D$ disabled, info b shows the breakpoint is set, but excution can't hit this bp.
2. boot up with I$/D$ dsiabled (in bootrom), info b shows bp is set and it could be hit. After this even if I$ is enabled in firmware more bps could be set and hit.
3. with option always_insert_bp could make bp work, didn't check memory contents though.

[abrodkin]

@ElliotCheng OK, so I guess what differs from the normal/default case is that combo: enabled I$ & disabled D$.
@anthony-kolesov @EvgeniiDidin does that ring a bell now?

[anthony-kolesov]

The case 1 implies that I$ is not invalidated properly. This is strange because this operation doesn't even depend on whether the core has an I$ - it is just done after the first write to the memory. (IC_IVIC used to be present in all ARC cores regardless of I$ presence, though some latest HS cores silently dropped support for that unwritten convention). It is strange, though, that always-inserted has an influence here. I think it makes sense to run openocd with a maximum verbosity level and compare the outputs between cases 1 and 2.

@ElliotCheng is the application loaded by the debugger or by the bootloader?