S3 Doesn't Have Folders!

The S3 folder delusion

When you AWS console you can create folders to group objects but this is just a delusion deliberately created by AWS to simplify the usage. In reality, S3 has a flat structure and all the objects are on the same level. Here’s the excerpt from AWS documentation that states this fact:

In Amazon S3, buckets and objects are the primary resources, 
where objects are stored in buckets. Amazon S3 has a flat 
structure with no hierarchy like you would see in a typical 
file system. However, for the sake of organizational 
simplicity, the Amazon S3 console supports the folder 
concept as a means of grouping objects. Amazon S3 does 
this by using key name prefixes for objects.

So essentially AWS is just smart enough to recognize the standard folder notation we’ve been using for ages to make this things easier for us.

All about S3 objects.

Comments

Post a Comment

Popular posts from this blog

Recursively Delete files of the Same Name in S3

  From the CLI you can delete all files recursively of the same name (/mydata.csv) using This is for cases where a file of the same name appears in subfolders of a bucket and you want to delete all occurences of that file. If you have bucket versioning turned on then you will not actually lose the file forever. aws s3 rm s3://mybucket/toplevelfolder/ --recursive --exclude "*" --include "*/mydata.csv" If you are paranoid about screwing things up there’s an optional – dryrun parameter that generates a report of what will be deleted, without actually deleting.   aws s3 rm s3:// mybucket/toplevelfolder / --dryrun --recursive --exclude "*" --include "*/metadata.csv" (dryrun) delete: s3:// mybucket/toplevelfolder /part_2/mydata.csv (dryrun) delete: s3:// mybucket/toplevelfolder /part_3/mydata.csv  
Read more

Python - datetime Formatting

Formatting datetime in Python can be a 'pain'. Formatting works in two directions - From a string to datetime. - From a datetime to string. The easiest way to remember how things work using the strptime and strftime is that: strptime (string)(parse)(time) strftime (string)(format)(time) Format a datetime to a string from datetime import datetime print(datetime.today( ).strftime('%Y-%m-%d %H:%M:%S')) 2020-07-22 10:07:16 Parse a String to datetime from datetime import datetime date_time = ' 2020-07-22 10:07:16 ' dt_object =  datetime .strptime(date_time, ' %Y-%m-%d %H:%M:%S' )
Read more

ORACLE RDS granting select privileges to users.

On the Oracle RDS service in AWS you have no real SYS/SYSDBA account or physical machine. So how do you grant select on SYS objects? AWS have the RDSADMIN account with some useful packages and I have used one of the procedures in RDSADMIN_UTIL to do the job. -- Granting Permissions to SYS Objects in AWS RDS -- In RDS you do not have access to the server or to a proper SYSDBA account.  -- begin     rdsadmin . rdsadmin_util . grant_sys_object ( p_obj_name => ' V_$SESSION ' ,         p_grantee => 'DASHVIEW ' ,         p_privilege => ' SELECT ' ) ; end ; /
Read more