Budget Committee - Regular Meeting

🎵 🎵 Thank you. Thank you. . . Thank you.

Thank you.

Thank you. The St. Paul City Council added committee to order. Roll call, please.

CM Coleman? Here. CM Johnson? Here. Committee member Dilworth?

Here.

Committee member Loster?

Here.

Committee member McVey?

Here.

Committee member Donnelly?

Here.

Chair Naker?

Here. All right. Welcome, everyone, to our June audit committee meeting. Happy summer. We have a packed agenda today. We have two items. One is the semifinal report from our last audit committee pending our thoughts and potential revisions. And then we have We're moving straight into topic review for our next topic. So we just never, never take a moment's pause here at Audit Committee. So we'll welcome back Dr. Britt to take us away. And like we were just saying, I think we're going to try to get 30 minutes, 30 minutes if we can. So we'll see what we can do.

All right, sounds good, fantastic. Well, thanks, everybody, and I appreciate your patience with us as it relates to our timeline. So I have a couple of intentions today. I'm gonna do the sort of walkthrough of the presentation that you had attached in the agenda. I'm happy to take questions as part of the presentation, happy to take broader questions, and then I'm just gonna invite you to continue, whoops, sorry, continue that feedback process, right? So written feedback that you wanna offer through Mia to us, We'll continue to take that. Mia has invited us today to have some specific conversations with her and with Greg and with Nicole and with some of the sort of city attorney team so that we can get some additional feedback as it relates to the report. So we're still striving for a June 30 update. final version, but I think if we need to push that, because we're still receiving some feedback to you all, if you're like, you know what, I can't make the 19th, but I can make the insert date that's between the 19th and the 30th, we'll continue to take that feedback and do that final full pass to make sure what you get is the final edited version of what we've got available. And as you know, since we're in long-term relationship with each other, we are here and around, so happy to continue to do additional work that you might need. So maybe I'll just open by saying a giant thank you to Nia, as always, and a giant thank you to Greg, who spent a lot of time with us walking through the GovQA system, spent a lot of time with us back and forth as it relates to folks that we might survey, did a lot of back and forth as it relates to just city rules about getting surveys inside the walls of the city system. So I think just want to say an incredible thank you. You all are a fantastic team for us to work with. So I'm going to go ahead and get started, but I will say, you know, pause me along the way as we are making our way through this. All right, so we're going to talk a little bit about background and purpose. Just a quick reminder, those research questions, the methods, so the kinds of things that we did to amass all of the pieces and parts for this. We'll talk a little bit about that GovQ8 platform itself, some of the things that we know from taking a look at that platform and taking a look at that data. We're going to talk a little bit about those conversations with the staff, conversations with peer jurisdictions, and I'll say sort of other folks, League of Minnesota Cities, Office of the State Auditor, the Data Practices Office. We'll talk a little bit about the data we have from the surveys that we did of department leads and of portal users. kind of the key findings and recommendations that we have, and then just invite discussion. And I think I'll close by saying, as always, we look to the audit committee to tune and pull up recommendations, drop recommendations down, do that kind of stuff. You'll saw we had a pretty light executive summary today. with a set of kind of eight major recommendations in kind of the back of the report. But again, look to you all to say, move this around, adjust this, we'd like to see this here. So really sort of looking for your directions as you're really setting the mark for what you expect the broader city team is gonna do as sort of a response to this. So you all know this. We were taking a look at that public data request process. We really had three kind of big questions. We had lots of subspecific questions, but kind of the three overarching guiding questions. How's it operating, that overall process? What's the experience of the public with that process? And what are the kinds of opportunities that exist to strengthen that process overall? And so we did lots of things. We touched lots of pieces and parts. And if you'll remember, we had a slightly larger scope. We took a look at managing the scope so that we could make sure that we're really considering things that would help inform and advise the city as it relates to next steps here. So first thing we did was took a look at as many documents as Nia and Greg could possibly find and send to us. We took a look at policies, training manuals, the internal workflows, and some model guidance from the state's data practices office. We did a literature review, as we typically do. That's kind of looking at peer-reviewed literature. It's looking at gray literature, kind of what's the best practice information that we have out there. We did have conversations with folks from Hennepin, Minneapolis, Duluth, the state auditor's office, the League of Minnesota Cities, and the data practices office. We also have information from the websites or the online publicly available sites from all of the jurisdictions that you asked us to take a look at. But those are the folks that we actually had sort of virtual interviews with. We did a survey of the department leads and portal users, and then we had a bunch of your GovQA data that Nia and Greg pulled for us, and we took a look at that at a really high level. If you'll recall early on, we debated how deep we were going to go inside that data, and so really trying to keep that up. sort of data snapshot at a high level, really just to take a look at turnaround time, what does that distribution look like, volume, those kinds of high-level questions that you were primarily interested in. And wow, that's a lot of data, 75,000 plus requests over this time period. All right, let's talk a little bit about just kind of the current process and the key data. So just the platform itself. GovQA is the CETI's kind of platform for receiving, tracking, managing all of those requests across all of the departments, right? So it's the backbone that the city staff are using. Its key capabilities, it's got that public request portal, so folks can sort of submit for requests. It's got the ability to be able to route requests across multiple departments. It's got what are called parent-child requests, so I can attach a big request to lots of baby requests and sort of keep those all together. I can manage some documents. I can think about redaction. It's got some dashboards, some dashboards that came kind of out of the box, and then some reporting that in particular some of your departments really had kind of built out in particular for some of their needs. The city uses what we would call a decentralized model, so common backbone, kind of common intake and tracking system, but really each department is doing the things that they need to do to be able to respond to requests in the ways that it fits the best for their particular department. And so you'll see a little bit in the survey what that distribution of behavior looks like across those departments. So one thing you'll see in the report, we did do a comparison just of capabilities. You'll see a grid in there with Next Request and Just FOIA, which I always think is kind of a funny name, Just FOIA, on kind of the other kinds of tracking systems and sort of management systems that are out there for possible consideration. So we do make a recommendation that if at some point it seems warranted, the city might consider examining alternative or possible other systems to use. These are at least a couple of the systems that are sort of best in class on the market. And so we just did a comparison of those sort of basic capabilities and workflows. But I think just a caution across all of these, you know, they really are about record tracking and management and they are not, I'm going to go into your system and pull stuff out and retrieve it. That is still a profoundly manual process. That is still something where staff are searching through emails and folders and documents and department databases. And that would happen regardless of the system choice that you made. And for all kinds of reasons, especially from a security vantage point, that probably makes sense. But I do think it's worth considering that it is inevitably going to be a set of hours and labor and expectation for city staff to have to navigate as they're managing through what these requests look like. There is no automated fix that is going to go in and find all this kind of stuff and pull it back. It just sort of really isn't available, at least through the mechanisms that are out there for public data request tracking and management. So a little bit about volume. So, you know, we pulled back all of those sort of 75,000 plus. We looked at 22 through 25. The 25 data was not the full year of data. And you know it often takes a while to wrap up the previous year's request. So we're going to see a little bit of a sort of lag experience as it relates to that 2025 data. So I think a few key notes. Growth year over year. Yes, you actually are seeing more requests come through. So I think the stress that the staff are feeling is a real stress. So that volume of requests has ticked up. You've got an average close time that actually has improved over time from kind of a 20-day average to about a 12-day average from 22 to 24. But you have got a really wide range for what it actually takes to for folks to actually be able to accomplish that close. So you've got a median of four days. Lots of stuff is pretty quick. And then you've got some requests that are lots and lots and lots and lots of days. So you really do have a volume of complicated requests that are taking the staff a pretty significant period of time to make their way through. So I would say that the data are distributed not in a normal fashion, if we were to think about kind of a bell curve distribution. You got about half a percent of sort of open requests, meaning you're closing really, really, really well, so there's not a big volume of those opens. I suspect you're hearing a lot about those opens, but in the volume, you're not seeing that giant

Just a question from Ms. Coleman. Quick question. Sorry if I'm jumping ahead. That is a shocking increase over three years. Yes. Question mark. Is that comparable to the increase that other cities are seeing? Do we have any idea of what's driving that?

So we don't have specific volume information from the other jurisdictions. But when we did have those conversations, and you'll see a little bit in here and a little bit in the report, everybody is reporting that pretty significant uptick over time, like just an increasing appetite for access to public available data. And every place, and Audrey was the one doing those jurisdiction calls, every place was like, yeah, we're watching the volume just jump and jump and jump. So it is startling. I think when we finally cracked open and looked at the data, we were surprised. But it was confirmed by the jurisdictions that we talked to. Yep, everybody is seeing that increase.

Can I ask a quick follow-up question? Yeah. Which is just, I'm assuming we picked 2022 as a random start date for when we were looking at this. Do we have any, I guess, maybe there was something there?

We recommended that because we came out of COVID during that time. So there was an uptick.

Okay.

And it's because. During the COVID phase of our lives, as you recall, everybody was working remotely. So there was a lot of interest about connecting with other jurisdictions. And that portal had a lot of interest. So we thought that was a good starting point. Interesting. Yeah. Okay. Thank you. Yeah. We can go back further.

Yeah. And you were fully online in GovQA, right? You weren't in the on-ramp period, right? You were fully stable inside GovQA and its utility.

Yeah. Okay, let's keep going.

Yeah, so you've got sort of a subset of departments with kind of higher open request rates, and I think it's simply because of the complexity of what's coming through. But again, we were looking at high-level count and volume information. We intentionally were not looking at the detail inside what those requests were, and that was really an agreement that we made as it related to our scope initially. As you can expect, police department is the lion's share of the requests, safety and inspections next, site administrator, fire. There's sort of a which department and then which type of requests are coming through. So there's sort of a couple of ways that we think about those variables.

Sorry, I just want to make sure we're reading this right. So the police department is 95% of all requests and the next highest is 2.2%? Yeah.

Mm-hmm.

Yeah. So it's- Good use of lion's share.

Yeah. Lion's share. Lion's share. Pretty much everyone.

Okay. Keep going. Sorry. Yeah.

Yeah. Yeah. You have seen in your human rights, your HREO area, some requests that have sort of grown year over year. So that's a bit of a jump. Yeah, and just kind of like I alluded to on the previous slide, you've got some complexity in some of the non-police requests that are coming through, which likely result in some of that longer close time. So I think kind of the takeaway, this is sort of the population perspective, improvements are going to have, especially improvements related to police response, are going to have the broadest impact because they are, again, the lion-lion share. But again, that citywide consistency, and you're going to see a little bit of that threaded through the recommendations, and we heard that in some of the department feedback, is going to matter across all of the departments, right? So you've sort of got this one big department, but given that you're seeing this volume increase, you know, kind of year over year, I suspect, you know, you're going to start to see that need across all of your departments, especially with those departments that have got those complex requests. All right, so let's talk about talking to folks. So just from a staff vantage point, effective, GovQA is a tool, effective for tracking, routing, communicating with requesters, kind of keeping that documentation. It's got some advantages. As I said before, it doesn't go do the job for you. It doesn't go retrieve the records. It doesn't search email, shared drives, all the kinds of things. That really is the lion's share of the labor that the staff are pouring into this. And then repositories that are kind of outside the platform. So repository that allows you to communicate with the requester. It does, GovQA itself, have some flexibility. And there are departments that have done a little bit of workflow work. Particularly police have done some workflow work. But that does mean that you've got some departments that have done some of that and some departments that haven't done some of that, some departments that behave certain ways, and some departments that behave other ways. So you've just got some inconsistent workflows across those departments. And then I think the age-old, it's got a lot of cool stuff hidden, but if you don't use it regularly or have the time to dig into it, you're just not going to be able to unleash all of those capabilities. So sort of that underutilized power of the tool, yeah, that exists. Let's see, kind of challenges across the system. So again, manual. Staff have to do a lot of interpretation, routing, splitting of requests. And a lot of stuff that comes in centrally is broken apart into different pieces and rerouted in lots of different ways. Those email requests are really resource intensive. So I want all the emails related to. That is a lot of volume of information that's got to be sort of pulled back and processed by staff in order to be able to be uploaded. And I think there was a sense of, like, we've watched all this volume increase, and we have not had a whole lot of increase in staff to be able to manage and process through this volume. So I think the Data Practices Act, I think folks feel like still is highly interpretable. It is not a zero-one, black and white sort of set of rules. expectations, rules, and so I think just that, that is something around which folks are sort of feeling. And then I think there's this sense that like folks who are putting in requests don't appreciate the amount of time that it takes for city staff actually to be able to respond to those. And while we want folks to have high expectations of the city and turnaround time, the volume of hours that it takes to put in to do that response and to do that close is pretty substantial. And so that sort of, I'm going to move over here, that sort of disproportionate sort of match in reality I think is sort of a challenge for folks. Some insights from the peer jurisdictions. So I think, again, back to this question, consistent themes, requests are growing in volume and complexity everywhere, especially those email-intensive requests. So those give me all of the emails, insert the rest of the sentence here. Again, reinforcing technology helps with that tracking, but most of the work is done outside of the system. Somebody said 90% of the work. in their particular area. And jurisdictions are primarily using GovQA, although there are jurisdictions that are using Excel spreadsheets to track. Yep, yeah. Clear written procedures define staff roles, regular training from peer jurisdictions, kind of the things that are really good differentiators about being able to kind of button up the process, make sure you're navigating sort of effective and efficient programs. That proactive disclosure, and I know we've talked about this, so stuff that could be released public, proactively being released public so that folks aren't repeatedly asking for it, is something that can sort of head off at the pass a little bit of this workload and just improve that public access more broadly. And I think just broadly, St. Paul's challenges are not unique. So this isn't a particularly exceptional set of experiences. This is the set of experiences that peer jurisdictions are having. And kind of thinking about that structured intake, how are folks putting in the request? And how are you managing workflows and department expectations are probably two of the things that would help sort of enable just some additional efficiencies. And so that's something that we heard from the peer jurisdictions. All right, so a little bit of sort of survey feedback. So we had 13 department leads give us some feedback. About two-thirds of folks are users for at least two years. Most folks are somewhere in the logging in weekly. They may be logging in less frequently. Most of the folks that were responding are processing requests themselves, and then they are also assigning requests to other folks. So it's a little bit of a both-and. that's happening. We asked them to rate the functionality of GovQA, about half said somewhat functional, about a quarter very functional, and about a quarter, you know, not functional. So sort of a distribution. The things that folks found difficult, understanding what's really needed in order to be able to fulfill a request, kind of unpacking that and having to figure that out. Thinking about deadlines and closing requests, so sort of just being accountable to that constantly, that constant list ticked queue that folks are trying to manage. And then kind of running the reports to try to figure out should I be looking at reports in order to be able to manage this, right? Like I think there's sort of a variable use across departments of kind of that reporting capability. What would help? Guidance on best practices and workflows. What should I be doing? Request intake form, so again, your public portal doing some work kind of amping up or elevating that, making that much clearer for folks to put information in so that it's easier when it shows up in the system for city staff to respond to. Automation, some of those kind of features, seeing if some of that could be enabled where possible. And then just kind of training. There was definitely this like, I got trained. And I could probably use another training kind of set of feedback. And I think just kind of the cross-department communication and figuring out who's responsible when a request has got multiple parts that spans multiple departments is just complicated. And everybody was like, that's just hard when we're crossing departments. It's just going to be noisy. It's going to be tricky. And that's going to be a hard part because we're going to go back and forth and back and forth and back and forth. Nia, I feel like I'm talking about a day in the life of yours. Again, and we talked about this last time, we surveyed a whole lot of portal users. We did not hear from a ton of portal users, but what we did hear is here. A lot of folks said it was easy to know which department to submit a request to. Most folks found getting into that portal pretty easy to do. Lots of folks said it was accessible in the preferred language. I will comment that that's probably a biased set of humans who were responding to the survey, but, you know, sort of we have the data that we have. About three-quarters said filling out that request was easy, and about a quarter said, you know what, it's somewhat difficult or it's pretty difficult. Improvements, and again, we offered to folks the option to select particular improvements. So this was not a, I'm coming up with my own idea. This was really in partnership with Greg and Nia. Let's be really thoughtful about what kinds of improvements we were going to offer for folks to select. So folks really want to know about their request status and next steps, right? So some clear information about that. instructions and examples of what to include so when i put in a request what should i be putting in i'd love a little bit more insight about that department roles so there's a little bit of kind of pick a department but like understanding what it is that the departments do i will say that's a little bit of if you'll recall the last project that we did inside out city speak versus folks who are coming to that portal and may not be as familiar with departments and all of the things that are really kind of insider language versus everybody across the city language. And then just kind of like simplifying what that request form looks like and some links to publicly available data. You're going to hear that theme over and over. If I don't have to ask you for it, it would be great if I could just have access to it.

Yeah. I see a question from Ms. Coleman. Sorry for all the questions. No, it's great. Yeah. So I'm surprised, given that 95% of the requests coming are for the police department, I'm surprised that 29% of our requesters extrapolating a little bit, but don't necessarily feel clear about which department they should be submitting to. Again, it seems to me like it would likely be obvious if you were submitting to the police department or not. Do you all suspect, understanding that we don't have a ton of respondents here, that 95% of people are very clear they're going to the police department, and it's that 5% that are like, is this DSI? Is this PED? Is it Public Works? Or do we think people are like, is this DSI or the police department that I should be submitting this request to?

I suspect a couple things are going on. I suspect it really is that 5% who are like, I am really not sure where this is going to go. I also suspect, and we had a sense for this, you've got some repeated requesters and you have some repeated requesters that are not about police. They're about other things. I suspect we also heard from those folks in this. Since we had such a low response rate, I'm like, folks who wanted to respond responded to this. I suspect that is a not a uniformly distributed set of humans who are responding here. So a little bit of a both and in that response. All right, so some recommendations. All right, first one, which I think, again, we heard, make better use of that GovQA tool. So there's some stuff inside of it. It's kind of a hemi, like actually enabling some of its functionality. Leveraging those features through training, shared practices, cross-department review, doing that stuff. So really making the best use of the current platform first. And you'll note at the end, before exploring the possibility of other platforms. So it's kind of like, make sure you're really maximizing use of what you've got hands on. Coming back around to that structured intake process, so really unpacking what that online portal looks like. Are there some things that you can revise? That public request form, more specific information, reducing that back and forth clarification burden, rerouting. Yeah, so thinking about kind of that front end, that input piece. Citywide procedures, again, you've sort of let every department make a set of decisions for themselves about what makes sense as it relates to that management. And so really considering the possibility of saying, you know what, across the board, we're going to have some consistency. So we're going to route the same way. We're going to document the same way. We're going to communicate the same way. And when we've got those lots of pieces and parts, either parent and child or multi-department requests, we're gonna behave in the same fashion when we manage those. Because right now it's really dependent upon who's kind of the primary actor for the first start, and then that is what dictates how the rest of that request is dealt with. Staff training and support. So regular training, some refresher sessions, some peer learning. You do have lots of folks that are touching the system, each of whom have some expertise in the thing that they've dug into. So really letting those folks actually get exposed to each other in a peer fashion, just to build that consistency, to build that confidence. And again, we did hear from folks who were like, yeah, I got trained by the city clerk and had my first half hour, and then I've been on my own since then. OK. So just a chance to be able to come back around and really create that almost community of practice for folks who are using GovQA, which might be helpful. Taking a look at that reporting data pretty regularly. And I don't have a sense for how frequently you're taking a look at it, but just that look, where are we at? Where are the bottlenecks? Where are we moving things through? Who's doing well and can we learn from? Who seems to be stuck and having longer clothes? And where do we need to potentially lean in and offer some additional support or some additional improvement opportunities? Back to the stuff folks are requesting frequently, make that easier to find. And I think that's both publicly making that easier to find as well as internally for the internal department staff. So within departments, stuff that we're going to get asked for a lot, or across departments, stuff that we're going to get asked for a lot, if it's possible to make those easier to access inside the walls or up on, you know, quite literally the city's website. so that staff are just not having to respond to some of the same things. And then I would say periodically take a look at whether or not technology is meeting the city's needs. So if your volume and requests grow the same way they have grown over the next year or two, then it may be at some point a place where you're like, all right, do we want to consider an alternative to GovQA from a system vantage point?

Let's see, public-facing information.

So in addition to the portal itself, just some of that, like, again, and I think Nia and Greg already knew this when they were starting to think through what kinds of questions we were thinking about asking portal users, thinking about amping up that FAQ, thinking about some more guidance materials, and thinking about just kind of status communications, just so folks know what is happening, right? Because I think a lot of the feedback you're getting is just, I don't know what's happening. So just some expectations about what that communication pattern should look like. And then there's looking farther down the road, can you allow for routing based upon subject, so actually enable that inside the system? Can you think about more proactive data that's released? So more and more and more of that disclosure that happens without having to have folks put in requests. And then I know, again, you're thinking about dedicated data practices staff, so what does it mean to have that, set that expectation And what load are they going to pull off of those department heads? And what is that balance going to look like? So that is the story that we have so far. You have a big report to make your way through with lots of data in it. You're also going to get a technical appendix, which we're assembling, which is essentially Everything that peer jurisdictions gave us that they said it was okay for us to share. And then we're just going to pull lots of stuff from the state just to make that available inside that technical appendix. And you can see the documents that are going to be in there in the back of this existing report. But we really wanted you to pay attention to the sort of primary data collection pieces. in the existing report itself. And again, happy to take feedback through Friday, happy to take feedback after Friday, and we are going to have some sit-downs with some of the team, city attorney staff, just to get some additional feedback about things that we should be tuning as it relates to the report.

Great. Well, a lot of food for thought. Thank you, Dr. Britt. I am curious to see, we want to make sure we have enough time for our next conversation. But I want to open it up to my colleagues in the committee just to see what questions or thoughts people have. I see Ms. Johnson. I see Ms. Coleman.

Well, thank you. Well, first, just thank you for kind of getting this to us so we can look at it. And I think from a procedural point, I just have a question around how far in the weeds you were able to get since 95% of the work happens outside of the the GovQA system itself, how far in the weeds were you able to get with just understanding what happens after the request coming piece?

Yeah, yeah. So what you're reading in the report is what we were able to get into from a weed vantage point. I will say kind of the sort of the cracking open those individual requests and seeing what they look like from a level down, from a detail vantage point, is not something that we did. And it's not something that we did on purpose, right? We really were kind of just trying to look at high-level volume types, trends, that kind of stuff. But I think we interviewed... Nia and Greg, and then we did sort of the department survey. So you can get a sense for, I'm telling you, most of the work that we're doing is outside of that system. So what you're seeing is about the depth that we got to without, frankly, unpacking every single one of those 75,000 requests at a level down, which we were really intentionally setting a boundary around, at least right now. This may be something that in the future you say, you know what, in two years, we'd like to look at this again and maybe go a little deeper. But you're in a particular scenario right now where we put some edges around that.

OK, thank you so much. Yeah, yeah, great question.

Ms. Coleman. Thank you. I will also echo the thanks. It's obvious that this is something that's taking up a lot of city time and resources. So having just more insight into the process and what's working and isn't is super helpful. And your last answer might have been the answer to this question as well. But I'm just curious about the recommendation to make frequently requested information easier to find. And looking and hearing and seeing that, a next step for the city might be figuring out what that information is that is frequently requested and then figuring out if we could and how we would make it public. Do you have anything else to share on that or anything else that came up in your all's research as to anything that people were bringing up consistently or that other jurisdictions mentioned they had proactively made public in order to try to work around some of those challenges?

Oh, that's a good question. Let me talk to Audrey about whether or not we heard that in interviews. I mean, I think we heard it at a high sort of general level. Yeah, as much as you can make public, make public. That's what the best practice literature says. As much as you can make public, make public. But we'll take a look at those interviews again and see if there's some specificity about like, hey, here's what we made public. Yeah, so we can take a look at that. And then I will say that potentially is... Again, were we to pull open those requests and do probably a qualitative analysis of those requests, we might be able to share back something like that. Again, we put some edges around this, but that is a future possibility. If you were to say, hey, let's take a look at a random sample so that we could get a sense for repeated requests, maybe generate a possible qualitative list of most frequently what of that could be public, what of that could not be public, right?

Thank you. I'm just curious, in terms of the 75,000, you weren't able to determine if they were discrete 75,000. I mean, some of them could have been repeat requests, or could you tell if they were all individual?

Yeah, they were, I mean, 75,000 individual. I mean, they could have been...

but they could have been repeat requests, right?

Somebody could have, yeah, come in repeatedly. Yeah, so, and we didn't, We didn't have enough information to be able to say we're going to deduplicate based upon anything, right? Because we didn't know, what we didn't pull was, I don't know who requested, right? We were really trying to sort of only have a sense for just sort of straight out volume, right? So, but at least from the staff vantage point logged into the system, it shows up as if it is 75,000 plus employees. individual, maybe not independent, but individual requests coming through, yeah.

And could you tell if there were like peak times?

I don't know that we have timestamp in there.

We didn't pull that record.

Yeah, I don't think we pulled timestamp, but that's a great... Question. OK, timestamp.

Yeah. Were you getting at time of year, time of day?

Time of year, I guess, really, or what was happening In the world, so to speak, as to what they were requesting, particularly since it's related to, it seems like, to the police. But I'm curious if there's duplication and ebb and flow.

Yeah. Yep. Yeah. Yeah. We really just, you know, we have a flag on year, right, inside that. So yeah. And again, we were trying to be, I think, careful about what we were allowed to take a look at and make available more publicly. But that's also a great, perhaps, additional, like, Are there some timestamp questions? Are there some qualitative, you know, sort of repeated public, you know, sort of ask kinds of questions that we could maybe do a next level down analysis?

Yeah. I think we are going to need to wrap here. Knowing that there may be other questions, I would really encourage my colleagues on the committee to reach out to Dr. Britt, reach out to Nia in between now and Friday. Send your feedback, in particular things that you would love to see presented when this comes to the council, like additional questions you want to see dug into. And Dr. Britt, I also think that it would be a good idea going forward to have, now that we have such a strong partnership with our administration, to have you come and do a presentation with maybe the council members and the mayor and administration after it's presented to the council, and maybe any relevant members of her staff. So I'm just thinking about that now. I want to say that out loud. So thank you so much. Oh, my goodness. Very impressive body of work. I have a ton of questions that I'm not asking right now because I'm trying to keep us on time. But thank you. Okay. Thank you. Thank you. Thank you. And with that, we are going to, and we look forward to having you back at council in a month or so. We'll see. Whenever we can fit it in. Sorry, can you pull your microphone closer to the answer? I'm sorry.

I think we're looking at July. Yeah. So we will coordinate, so July 8th was the date we had originally discussed as a possibility of presenting the council. Great.

All right, we are going to pivot to our next topic, which is new topics. Nia, do you want to take us away?

Actually, I'm going to pass over to our advisor, Ross Sutter, to kind of walk us through There were four of us who met to discuss and rank the score. But I think everyone agreed that Jeremy would do a nice job.

OK. Well, thank you, Nia. And Chair Naker, as Nia said, four of us met. I guess you could call it our risk assessment subcommittee to go over the list of proposed projects. So there were five of them. And I think the key takeaway is, They're all very good audit topics. They all scored very highly on the scoring matrix that we used. And so that affirms we have good topics. Any of them would be a good topic for an audit. So just leading off with that and then getting into the details. So first off, I think what we propose is combining two of them, which are the business licensing and building code operation audit topics. And the reason for combining these is because they're both within DSI, and my understanding is they make use of the same system, which DSI is using a new system right now. So it might make sense to combine those. And so then the other topics were college savings accounts, district council reform, and street maintenance and potholes. So we went through our rubric scoring matrix, which you have copies in front of you. And so we have rankings of low medium and high for six different questions and We had two of the projects tie well actually we had two ties so the top two are scoring topics were the DSI business licensing and building code operation, which I had mentioned first. But with a caveat that because of the new DSI system, we would recommend maybe that being the last topic audited in 2027. So I think what we're looking at is maybe the remainder of 26 and then also looking at calendar year 27 and for scheduling these audit topics. And so we kind of looked at maybe there being three slots for timing of putting those in. And so with that being the case, potentially because of this new system and the learning curve and getting that all in place, that that would take maybe the third slot would be our recommendation. And that scored the highest but tying with that project was the college savings account project and that one I think mainly scored highly because of the backing of the mayor's office and the interest in in doing that audit to look at that program, kind of a performance evaluation. Is this program running as intended? So that would be our recommendation. the next one so those two are tied and then the final two were also tied but with only one point difference so we nearly had a four-way tie but then the final two then were district council reform and street maintenance and potholes so in thinking about if we have three slots it might be difficult to decide which one of these four projects now to drop. I guess it would be a choice between district council reform and street maintenance and potholes, at least as it comes from the recommendation from the risk assessment subcommittee. So any questions about what we did? We met, it was last week? Yeah. So it was community advisors, McVeigh, Dilworth, and then Nia and myself.

Well, thank you for the report, and thank you for the work. It's probably the most important part of our job is to figure out what we actually want to look at, and this is a huge part of it. So thank you for doing this. Just one clarifying question before I open it up to others. So I'm not seeing the shading on the business licensing one. But I'm assuming that it was shaded all threes. Questions from my colleagues or comments? And again, our goal is to choose three topics today to recommend to the council. Would also want to, as usual, make sure that we're doing due diligence with the relevant department directors and staff on each of those topics prior to bringing them to the council. It was great to be able to speak with the mayor, but we also want to make sure we understand from the departments that there will be buy-in to these topics. But for today, we want to get down to three. So I'm looking to folks for questions and comments. Madam Chair. Mr. McVey. Thank you.

I will start off just by saying how thrilled I am that we have these topics, because I think that residents really did such a wonderful job this audit cycle. And I'm really proud to be a resident of St. Paul, because how many people actually came to the district council meetings? shared their ideas and their concerns with us. And I think that's really valuable, especially in this day and age when there's a lot of eroding confidence in our public institutions. And I think what we do here on this committee is really important in terms of demonstrating that we're all about government accountability and improving public service delivery for the folks that live here in St. Paul, all 300,000 plus of them. So I have a couple comments. Regarding district council reform, again, this is from the district councils themselves, from their staff, and also from the officers on their boards, and so it's important to keep that in mind, that they're gonna have some motivation behind this, is what I'm trying to get at, but still, they really would like us to, they would highly encourage us to consider looking at their organization, their structure, their finances, how they operate, their core services, because a lot of them are struggling financially right now. in conversations with some of the executive directors or some of the district councils, they are having some difficulties, especially with like indirect costing with PEDS still. And I think you know who I'm talking about, Madam Chair. And there's also a few, it's not just one of them though, it's actually multiple that have expressed this concern. And I will say that with district council reform, I think given how many of the district councils brought this up, in terms of the board members and their staff. think that if we don't do this within this audit cycle, it's going to create some problems in terms of our relationship as a committee with the district councils going forward in terms of our ability to work with them to bring the public in and get feedback for future audit topics, simply because they may say like, oh, you brought this idea. We really want you to go with this idea, and you brought it to the full attention of the committee. And they may just be like, we don't want to really go through with this process in the future. Because they are our eyes and ears on the ground here in St. Paul. And so that's just something to keep in mind. I'm not saying it necessarily needs to be maybe this year or 2027, but it's definitely something we need to look at at some point, whether it's in the next three years, if we don't get to do it this year or next year. So I just want to make the committee aware of that in particular. The other thing is with college savings accounts. I'm sorry. I'm starting to lose my voice. I'm getting cold. So bear with me here.

Just getting emotional about the work of the Audit Committee.

Oh, yes. Absolutely.

That's what that must be. Happens to the best of us.

I'm so moved right now. With the college savings accounts, I know this is a really big, this is something that is being proposed. The mayor, she is a proponent of this, because I know she had a hand in creating this program years ago. I will say this, there is a lot of information out there already about how the program is working. There could be more information, because the problem is we don't have depending on what comes out from the Office of Financial Empowerment, there's different metrics that they're using based off of how you frame the question. In terms of participation, and then also how many, because of the fact we don't even automatically enroll kids when they're born here in St. Paul into the program, that creates some difficulties too, which is unusual, because most programs are automatic enrollment, but I'm getting beyond my wheelhouse here. That being said, part of me is wondering if that topic I think there's merit for us to do an audit of it. I do think that. But I also wonder if this is almost something that could be evaluated during the budget process as well, regarding that program. And that could be something that had between the mayor and her administration and the council. Again, I'm just putting these ideas out there. I'm just kind of thinking aloud at this point, trying to determine what makes the most sense for us as a committee to be doing. And I'm just initiating that conversation at this point. So I'm done rambling, Madam Chair.

Thank you, Mr. McVey. Never rambling. Other thoughts? Mr. Donnelly.

OK. First of all, great work by the team, sincerely. We have four awesome topics. I love the rubrics. Taking all the information, getting it to this point is a real achievement. Thank you for doing that. So wherever we land, it's a good outcome. Okay, so when I worked in a factory, we would do lean events. And we would sort of use criteria to decide what we would work on to improve. And one of the key points would be like, if we did a lean event on this part or this process, could we extrapolate those findings to different parts and processes so that what's gained is not just in this one thing, it's over here. And that's kind of where I'm at with College Bound, where it's like my counsel to the city is to say, If you're just going to look at College Bound, and that's the only lens you're looking at this, it's probably not a ton of value. But if you're going to say, we really need to look at programs across the city, how we fund them, how we evaluate performance, grant management, and we are going to take the findings from College Bound and use it as a basis for improvement across the city for all our programs, then I see a ton of value. And that's a question whether the city, how you're approaching this, what you're thinking about programs. And so that would be my counsel to say, think about if there's a genuine interest to learn about program effectiveness and integrity, and whether there's the bandwidth to take these findings and put it across the city or not. And I think that's some of the deciding factor, whether you see value in that topic or not. But again, good job here. These are all great topics, and I think you'd find value going for any of them, really.

Thanks, Mr. Donnelly. Ms. Johnson.

Thank you. I really appreciate just kind of listening to everyone's feedback. And also to the folks that took the extra time to go through the scoring and to actually review the topics and give your thoughts on your rubric scoring. I think that that's also really important because we have to kind of see where folks land. I know during the last meeting, we talked a little bit around our own scoring. And so I'll just simply speak from my vantage point. I think for some of this, I like to think about the next steps and what we do with the information that is given to us. I'm unclear what the audit, for the college savings accounts piece, I think I'm a little unclear reviewing whether or not it's doing what it's intended and then trying to understand where we would go from there if we found that it was or even if we found that it wasn't. And I just, so I'm a little like, I'm kind of like what would the outcome create and also what would the outcomes be and that would be something that I would really have a, like, I think we should, if we were to evaluate the college savings accounts, I think we should have a clear understanding of what that means for the findings. And especially because there is an opportunity, I think, being that I believe that it sounds like it was, you know, the mayor's request. I think that would be what I would want to know is, like, to what, not to what end, but kind of to what end? What exactly are we aiming to look at and what we're looking for? Because when I look at the description, it's like, what does equity mean when it comes to the effectiveness, the equity of the program? Of all of the others, they're very like, look at the organization, the core functions of the district councils, and looking at the forming formulas. For the building code operations, there's also kind of like that specificity that I know who they'd be talking to, where they'd be going. With the program evaluation of just like the economy, efficiency, effectiveness, and equity of both college-bound and college-bound boost, like that to me requires a little bit more of a follow-up. So I respect that it had higher scoring, and I think to the committee's reviews, but I think... I think that if that's the decision that we go with, it needs to have a little bit more questions asked. Because I think the description as written is subjective, not research-based. And that's a little concerning for me. And then also on the flip side, I'm just like, and depending on what the findings are, I'm just like, well, what does that mean? Because it's a program. It's tied to multiple funding streams. In my eyes, it carries actually a little bit more risk on the operational side, on the financial side, on the compliance and the reputational side. To me, those risks matter of what the findings would be if we're not confident in the findings themselves. And so that would be the question that I would ask for probably for the administration and also for folks that are looking through this topic. Because you just... My brain goes into we have multiple external funding sources falling into this as well. And I just would want to make sure that we were setting it up for success versus literally creating a public-facing documentation to ask questions and things of that nature. That would be the only thing of concern for me of that topic if we moved forward, if that makes sense. Yeah. Thanks, Ms. Johnson.

Ms. Coleman, and then I'm going to move us towards we have to drop one. So move us towards that.

Thank you. I'll make it quick, and I will give my proposal for when we drop. Super appreciate the subcommittee that put this together. It's really helpful, I think, to see that all of our topics are strong and we think could sort of meet the basic requirements that we have set out. To my mind, the DSI one is a no-brainer. as is the street maintenance potholes for reasons that I think are articulated at our last meeting, but those are core functions of the city. There are things that we need to do really well. There are things that we spend a ton of money on. We hear about them from residents every single day. I think we'd be very remiss to not move forward with those two audits. So to my mind, it sort of leaves the question between college savings accounts and district council reform. I would say I would be a vote for the district council reform as the third topic, I think both as a mark of appreciation, like Mr. McVey was stating, of how much work the district council has put into getting us feedback on this. I share some of the concerns that Councilmember Johnson raised about the college savings accounts. The other one that I just think about a lot is we put a lot of trust in our district councils. That's reflected to some extent in our city funding of the system, but not necessarily entirely reflected in that since we don't give them very much money. But we put a lot of trust in them to be the voice of the community. And I think that they have a great deal of influence in our city. how they work, how they're currently working, how we can support them, how we can take advantage of this quite significant role that they play is to the benefit of everybody in the city. So those would be the three that I would be supportive of moving forward with.

Great. Thanks, Ms. Coleman. Well, I think I'm going to... open it up to, assuming that's not a formal motion, but sort of a proposal, to see what others, I think it mirrors some of what we've heard around the table, and I'm curious how others would respond to the idea of having our three be, dropping college savings accounts, essentially, and having the other three be the ones that we move forward with.

What's your opinion, Council Member Nagy? I'm curious.

I actually, would rank, I would agree, but because I would rank the first parameter a little bit differently for college bound, I think it's funny. We were interested in that topic in part because we were interested and then the mayor was really interested. I think as a result, you want a sort of sweet spot. You don't want the mayor to be zero interested. But in some ways, because the mayor has expressed interest in evaluating this program and has, I think, also conveyed that to her own staff, And because there is so much data available, I think the likelihood that, I think two is actually a more accurate rating for this one. Topic has not been studied within the last five years, but city has plans to address the topic. I think it's more likely than not that that will be looked at. And I think some of these other topics, maybe not so much. I also had not thought about the responsiveness to the district councils wanting to be studied. I think I've always thought about this topic as something that would not maybe be welcomed by them. But as our community engagement chair is attesting to, and also maybe sole member of the committee at times, the subcommittee, I think that is really, that sits well with me. So I would support the proposal. Mr. Lastetter.

Yeah, Chair Naker, I think that's a very important point. The risk ranking, if there are plans in place already to address the college bound, that would change the results of our scoring on that, which we were not aware of at the time. But yeah, what I'm hearing around the table There's concern about the scope of that engagement. We'd be need to be very careful about what that is I love the suggestion from advisor Donnelly about you know could it be looking at Citywide programs as a whole like best practices. What's working? What doesn't what you know are we gonna? Do similar programs in the future what lessons learned could we apply to those programs? That could make sense But, yeah, the scoping would be key, and if we would move forward with that, to Councilmember Johnson's comments, you were saying what questions do we want answered. Those would be the draft audit objectives. Yeah, what questions are we wanting to answer with the engagement, and that would be a key piece, I'd say.

Um, hear you and agree. And, um, and by the way, I totally would echo that the committee, the subcommittee, I'm not sure that the city for sure has plans to address the topic. So it is a little bit of a, maybe 2.5 on that one. But I think just given what I've, what I've heard the mayor say, I'm feeling like it might be more likely than not. So, um, no balls were hidden from the subcommittee by any means. Um, I also just want to note that the three that we're talking about advancing, um, are all. pretty hefty. And so we're going to have to have some difficult conversations as a group about scoping. For example, district council reform, revisit the financing organization core functions of St. Paul's 17 district councils. So I think that is not to say that we shouldn't advance these. I think as a group, we could make an argument that it's really worth doing all of these in full, and it just might take more time. And we're talking about phases of each of these. rather than just trying to take a small bite off of each one. But we're going to talk about that. I doubt that we are going to be a three or four month study all of street maintenance, or all of business licensing, and building code, and all of our district councils. That's going to be six to nine months each. I could easily see. And obviously, we need to talk to Dr. Britt about that. But I just say that as a caveat as we consider these.

It may take us through 2028, actually, potentially.

which would mean fewer topic selection meetings for us. That's good. But yes, Ms. Dilworth.

I just want to, and I've been listening to the conversation, and I think I want to clarify when we had the conversation about College Bound, and that doesn't mean that I'm trying to say that we should be doing College Bound. And these are, the description is very high level, but the conversation that we had was a little more detailed. And it's unfortunate that it didn't, and I'm not saying anything, but it's unfortunate it didn't make its way to this document to explain kind of what we were thinking. Because as I recall, when we were talking about the economy, is it sustainable? So there was different top things that we were looking at. With equity, and even Mr. McVeigh said, it's not something that, If you're born in St. Paul, you automatically get it. That's the equity. Because if you have to apply, that's inequity on many levels. Because there's a lot of people, parents, that won't apply. And it's not because they don't want to. They may not. It gets past them. There's different reasons why people don't apply. We have all sorts of scholarships out in the world that people don't apply, students don't apply for. So if this really was what they wanted to do, then they would get it automatically. As soon as you were born in a St. Paul hospital, whatever, you got it. And so then, and effectively, an efficiency. I think there was a little more depth to the conversation that we had just in terms of what we were thinking. But I also, as Mr. Donnelly was talking, was saying, I don't know if it's a full audit that needs to happen for this particular data. program, and I don't know if I'm splitting hairs, but a review of some sort that helps to understand where it is. And if that's something that we can talk to the mayor about them doing that, I think that would be great. Because I do think there needs to be something that answers some of these questions that the two gentlemen are even bringing up and that we had when we had the discussion that if we just say we're not going to audit it or not have it on the audit or move it forward, it may not get answered. That's just my point.

So first of all, Ms. Dilworth, I really appreciate that point. And I think there is a middle ground between we're auditing it and we don't think it needs any consideration whatsoever. i wonder if maybe there is a communication that comes from this body for example to the mayor especially given that this topic came up you know in our discussion with her to say um i think that would be actually really good closing the feedback loop of this is we ended up not deciding to audit this topic for these reasons that don't mean we don't think this program needs attention um and i really appreciate the points um that mr donnelly made and mr lasseter echoed about and those are likely things that could be considered applied to future programs or current programs when we're deciding whether or not to start or continue existing programs, not just this one. So if the committee, if we do decide to not move ahead with that audit, I would certainly support some sort of communication to that effect. And maybe even, and I want to recognize that Director Owens is here who directs the Office of Financial Empowerment, maybe even a conversation with the team as they are getting into the role and looking at the program about what they're finding, what their plans are. That's something that this body can certainly invite and engage in. And we've done that in the past, inviting department directors to come and talk to us about programs separate from auditing. So, Mr. McVey.

I just have a quick question, Madam Chair. Is it possible that we could work in concert, since we have the director here? Hello, ma'am. Could we actually work in partnership with OFE? Perhaps they do a review and they come back to us as a committee after the administration does their own internal evaluation report back to us. And then if we say the council needs to make some legislative changes or whatever to the program. I don't know if it's set up in city code or not. I imagine it must be. Correct me if I'm wrong. I don't know. Like setting up changes to the... Like if there needs to be any reforms to the program after having that kind of dialogue with the administration, if they do their own internal review and then they come back with their evaluation to the committee and what have you. Because I think Ms. Dilworth brings a... a valid point and we did have a conversation about like for instance like the fact that how the program some of the account i think how the accounts are structured versus like um how like under um with section 529 and all that too so was that part of our conversation i can't recall for freshman was that we had we had a robust conversation about this particular

Like 20 minutes, I think, right?

We were all over. Yeah, we were all over.

Can I ask a clarifying question? Yes.

Just a quick clarifying question, I think, just so I can know where to... The timing for... We're picking three topics because we feel like we have the timing to complete it with the topics that we have, or do you get... Could you just talk about how we're picking the topics, how many topics? And then also, is that because we think that that's something that we'll be able to get done in the course of the end of 2027? Is that why we're picking three at this time?

So great question. We typically complete two audits a year. And rather than spending a lot of time choosing just one topic when we often have one very close runner-up, what we decided, as we're always looking to continuously improve and become more efficient even as an audit committee, was that we would choose two topics at the same time. So we have one selection process that yields two results. One takes about half a year. The other one takes about half a year. We are inevitably running a little bit behind schedule. And so rather than being at the start of 2026 right now, these are becoming sort of the end of 26, beginning of 27 audits. The idea here is to send three recommendations to the council, have the council choose two, which would be the next two. And as I was saying earlier, maybe not even the next two for the next year, but two that might take even longer than that. But that's the thinking.

Okay, I think with that, then my follow-up comments would be, I have a lot of questions still left around the college savings account, and I would hate for it not to be able to be considered at a later point in time when it comes to a follow-up conversation. I have questions that I'm not sure. Typically, I'd be like, come on up to our chair and have a conversation about it. But I think I have a couple questions about just really where they, to be clear, and also... With this particular piece, I think the question for me is more if this is around program evaluation of the actual program itself, or if it's around auditing what has been done with the intent of correcting or taking the information that's given to do something with it. And I think that they are distinctive things. And that's actually part of my hesitation around this topic. And so I'm supportive of what is being considered. I do think that those three topics together, like if I had to choose between the potholes and the DSI piece, the building code piece, because those are two substance, the college savings accounts for me would make the most sense. Because I don't think it would take as long as the district council reform is going to be. those three together and knowing that we're only doing two i'll just say there are three heavy ones that probably will take a long time like going through a system that's been in place for decades is going to take a minute and so i just realistically don't know if they'll be able i would really value dr britt's um input on just what she thinks will be like realistic tangible pieces by the time this gets to council because that i mean i am curious to know how long i think it's going to take to go through 17 systems um And anticipated taking quite a bit of time. So I'll just offer that and simply be done. And those will be my follow-ups for the administration and for the director of what would be the goal of this study. Because I see the description, hear the conversations. That matters to me about whether I would want to, depending on what the results would be, I could... That would maybe change my vote today.

So I have a crazy suggestion based on what you just said, Chair, especially because we get to make our own rules in the audit committee and set our own schedule. We could also, rather than deciding between these today, we could do the follow-ups with the department directors that I mentioned before we make a final decision. And that could include talking to Dr. Britt about each of these topics and getting her estimate of what we're really talking about in terms of a scope. and timeline, and we could then come back in July with all of that information and decide, based on all of that, which are our three. Is that something that would help people in making? I don't want to rush unnecessarily. And I'm seeing some heads nodding, and I'm seeing some.

OK.

I like process.

I mean, as long as we're going to do that consultation anyway, I think it does make sense to kind of have that. Why not? And I would also just say that to the extent that the subcommittee talked about things that really weren't reflected here that you want to share either in an email afterwards or just other considerations about specifically college savings accounts, for example, feel free to do that. This doesn't have to be the only record of the conversation that was had. Mr. Lastetter.

Yeah, Chair Nacre, I think we were kind of swirling around a new idea, too, that I didn't want to get lost, the idea of maybe willing departments doing a self-assessment, that maybe that could be an option. We could develop the questions, kind of like what I was saying before, the audit objectives, questions this committee would like answered about a program, and if a department is willing, and has the capacity, they could do a self-assessment and report back. And that could be a deliverable. And then this committee reviews those and decides, is that sufficient? Would this be ripe for an audit in the future? But maybe that's, you said we're an innovative committee. That could be something new that we look at as a possibility.

Yeah, so maybe sort of an audit light of a self-assessment, and then that frees us up to do something else, but still provide that support.

Still look at a worthy topic. To worthy topics.

But okay, I like that. Thank you for bringing that back, Holmes. We did not lose it.

They do that at GAO, actually, Madam Chair, with federal departments on sub-programs within federal agencies at the Government Accountability Office. That wouldn't be a bad idea to actually adopt as a practice here at the Audit Committee for our city. because we could actually have a more thorough understanding of things going on in city departments on a regular basis. And that would provide us more information as well.

So I'm hearing, I want to wrap this up, this is great, that we are going to, no, this is, it's all good. We are going to talk to departments and Dr. Britt about these topics. We are also going to raise the possibility of a self-assessment sort of in the guise of were you already planning on doing this anyway or would you like our support in developing some questions and doing this on your own without outside consultants doing it. And then we will bring that back in July, and we will have a real decision before us at that point. Sound good? Thanks, everyone, for the robust conversation. Sorry we went over, but I think it was worth it. We are adjourned.